KMesh

The idea: coordinate reuse around existing engine sharing

Imagine an agent reads a document and then launches four follow-up requests. These related requests form a family. Their common context is the ancestor; each follow-up adds private input and output. The model saves attention state for text it has processed; this is its key/value (KV) cache. vLLM runs the model, and its existing prefix cache can share that saved state when requests begin with the same tokens. LMCache stores state outside the engine, in host RAM or on storage. Singleflight makes one request restore a cold ancestor while matching followers wait and then adopt that copy.

Sharing inside the engine leaves another question: can the external cache serve later lookups without reading the SSD again? External retention keeps the restored objects in LMCache, in host RAM or an external GPU arena. Application notice can then move preparation before the follow-ups arrive. KMesh investigates this coordination and the lifetime of shared inference state.

Cascade attention and family-aware offloading

FlashInfer's Cascade attention shares access to a common KV prefix while computing attention for several queries. It computes their private suffixes separately and merges the attention results. This reduces repeated GPU memory access inside the attention computation.

KMesh investigates the lifetime and movement of reusable state: which copy to retain, which requests can share a restoration, and when to prepare it. Cascade can consume shared KV after it becomes available; family-aware offloading can also coordinate artifacts with no attention computation at all. Family membership supplies reuse hints, while compatible object identity determines whether sharing is valid. The shared-ancestor structure connects these efforts, but their mechanisms have separate costs and correctness conditions. Their combined performance benefit remains unmeasured.

See the Cascade controls and retained early-loading comparison for the measured results.

Shared artifacts beyond KV cache

An agent family can share more than attention state. Several children may need the same retrieved documents, document embeddings, tool response, saved memory or execution record. The shared part can remain immutable while each child owns its changes. KMesh's broader goal is to coordinate the loading, retention and release of those artifacts through their existing storage and restoration interfaces.

The agentic cache-offloading model explores how richer artifacts, indexes and copies can grow the working set beyond RAM. It is a capacity projection, not a measurement of KMesh performance. Small plan templates can fit in memory; large shared artifacts create a different storage problem. KMesh should reuse the existing artifact loaders and coordinate with their development. CPU-consumed artifacts need an appropriate host-storage path; DMA-BUF is relevant where a qualified device-buffer path helps their actual consumer.

The objective is to establish whether family information makes offloading more useful. A fair comparison keeps ordinary caching, the storage path, memory budget and output requirements fixed, then measures what changes when the cache knows which requests are related. Fewer reads are useful evidence, but completed agent work, waiting time, retained memory and displaced cache hits determine the tradeoff. The KV results below establish bounded benefits from retention and early preparation. They do not yet establish a general family-policy benefit or non-KV artifact support.

What the results establish

External retention completed the read-once path while preserving the singleflight sharing that was already working. In the earlier preloading comparisons, singleflight was on and the children shared one ancestor inside vLLM. LMCache lookups could still trigger repeated SSD reads. Keeping the external copy removed that duplicated storage work.

The retention comparison measures keeping the external copy. The latest comparison measures when to prepare that retained state. Together they show the value of singleflight plus external retention, followed by the additional benefit of advance preparation. The last question explains the sequence and full timing results.

The percentages measure waiting after the fixed scheduled arrival, comparing early preparation with loading that starts when the requests arrive, with the external copy retained in both cases. Singleflight and read-once storage are enabled in both controls. The test covers one small model, four or eight requests and eight output tokens per child. Late submission stays in the waiting time; the advance preparation and its one warm-up token remain real work recorded separately.

Reusing shared text at new prompt positions

The retention results above share an identical beginning of a prompt. A separate experiment asks whether four follow-ups can reuse saved state for the same text when each puts different instructions before it. Moving the text changes its position and preceding context, so copying the saved bytes alone is insufficient. Our TokenDance-style implementation extends LMCache's CacheBlend code: it supplies the shared segments and their positions, recalculates missing positions, and uses CacheBlend to select additional work before reusing the rest.

CacheBlend already supports non-prefix reuse. The TokenDance-style case adds an explicit group of related requests and identifies their shared text segments for KMesh. These tests compare our loading and coordination changes; they establish no advantage over stock CacheBlend's repair method.

We test loading 416 shared tokens through a separate cache process and running that same repair algorithm. The four follow-ups generate exactly the same tokens as when that extended TokenDance-style method runs inside the model process. Their saved attention state and next-token scores match bit for bit, both when finishing one follow-up at a time and when running one model layer, or computation stage, for all four follow-ups before advancing to the next layer. The group uses LMCache's cache in host RAM. Its failure test lets three follow-ups finish while the failed one cleans up and retries by recalculating its whole prompt.

A separate SSD comparison runs follow-ups one at a time. DMA-BUF, Linux's buffer-sharing interface, lets the SSD read directly into GPU memory instead of staging the read in host RAM. Read-call time falls by 3.7% across seven matched pairs, but complete restoration is faster in only four pairs. In that comparison, checking the loaded data still copies it back to the CPU. This storage comparison has singleflight off and fixed retention on; it explicitly evicts the retained objects before every child to force fresh SSD reads. The earlier read-once experiment kept retention and singleflight on across the children. Combining this storage path with the group remains untested. The recovery page explains the setup and results from the beginning. These runs all use CacheBlend's selective recalculation; we have not shown here that it matches recalculating the entire prompt or improves serving speed.

A follow-up keeps DMA-BUF reads fixed and computes the same SHA-256 checksum on the GPU. It returns 128 digest bytes per child instead of downloading the 12 MiB source payload for checking. Our tested CUDA kernel is much slower than CPU hashing, and complete restoration is slower in all seven matched pairs. Model outputs still match exactly. That experiment retains CPU hashing as its default; this result concerns one kernel and the existing whole-object format. The GPU checksum results and reproduction instructions give the measurements and the next optimization targets.

Explore the research

Each detail page covers five research topics with the experiment, findings, limits and reproduction instructions. The overview gives the idea and conclusions; the detail pages retain the full measurements.

Papers

• Cascade Inference: Memory Bandwidth Efficient Shared Prefix Batch Decoding
  Shared-prefix attention computation and merging with private suffixes. The post describes multi-level prefix hierarchies already implemented in MLC-Serving; KMesh explores coordination and future reuse around that computation.
• DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference
  Cross-turn temporal KV reuse in long-context, short-append agent trajectories, and a dual-path loading scheme that rebalances the resulting storage-bandwidth bottleneck.
• TokenDance: Scaling Multi-Agent LLM Serving via Collective KV Cache Sharing
  Round-aware segment indexing, collective position-independent KV reuse, and Master–Mirror diff-aware storage for sibling agents in synchronized All-Gather rounds. The design KMesh reconstructs as its open baseline.
• ForkKV: Scaling Multi-LoRA Agent Serving via Copy-on-Write Disaggregated KV Cache
  Fork-with-copy-on-write for multi-LoRA agents: a shared trunk plus lightweight per-agent divergence. The evidence that the shareable unit extends past the KV cache to adapter-affected state.