Reusing shared context across processes and storage

21. Restoring state into vLLM's GPU cache

Before running the model, we verify that restoration places saved state in the right vLLM cache blocks. The GPU checks exercise our destination writes and position adjustment using allocations managed by vLLM.

vLLM divides its saved attention state into memory blocks. Those blocks can store tokens, attention heads and key/value data in different orders. An attention head is one of the model's parallel attention calculations. The restore code must find the right addresses for each supported arrangement.

Position matters too. The model encodes a token's position by rotating its key vectors, a method called rotary position embedding (RoPE). When shared text moves, the restore code adjusts that rotation. This adjustment handles position; it does not by itself fix the effect of different preceding text. The real-model comparison checks the additional recalculation needed when the preceding context changes.

All 32 required synthetic GPU cases passed on October 23. They check that values stay unchanged, moved keys receive the expected positional adjustment, and memory outside the requested destination stays untouched. These use small, constructed arrays rather than a language model, so they establish correct copying and addressing, rather than answer quality or speed.

22. Owning shared state across processes

Sharing saved state through LMCache's existing cache process requires clear ownership: a request must be allowed to use each allocation, and that allocation must remain alive until the request's work finishes. The tests check these rules across the process boundary.

The model worker and cache service run as separate processes on the same machine. The cache service holds the shared originals while individual requests come and go. Before it accesses GPU memory, the worker registers it: it supplies the memory handles, describes their layout and establishes which client owns them. The service must distinguish that exact allocation from a later allocation at the same address.

The tested service keeps ownership tied to the actual client and allocation. A replacement client cannot reuse an old registration merely by claiming the same instance number. Closing a registration stops new work, but its buffers remain reserved until already accepted copies and GPU operations finish.

Consider a cancelled request whose copy is still running. If its destination were immediately handed to another request, the old copy could overwrite the new request's state. The service therefore waits for physical completion before releasing memory. Losing a heartbeat or cancelling a request is insufficient proof that the GPU has stopped.

23. Completing GPU copies before memory reuse

Requests can fail while a GPU transfer is still running. We test that the recovery code waits for accepted transfers before releasing memory, including when the first cleanup attempt fails.

The memory stays reserved until pending work has actually finished. GPU copies can run asynchronously: the call that submits a copy may return before the copy completes. The service must also wait for the model's earlier writes before reading their results.

Six tests exercise real copies between GPU memory and host RAM with separate processes. Each direction has a successful case, an error whose pending work can be completed, and an error where the first attempt to wait is refused. In that last case, the source and destination remain held until a later wait succeeds. All six passed.

The implementation calls this wait a drain. Draining finishes already submitted work; it does not mean submitting the failed operation again. This is what makes it safe to release buffers or recalculate a failed request into them.

24. Evaluating model outputs and DMA-BUF loading costs

For each request, the generated tokens match when loading inside the model process and when loading through the separate cache process. Their saved attention data and next-token scores also match exactly. DMA-BUF gives a small, consistent reduction in read-call time, but no consistent reduction in complete restoration time. These are separate comparisons.

Model and workload

The model is Qwen2.5-0.5B-Instruct, a small model with 24 layers. A layer is one stage of the model's computation; each request passes through all 24. One leader request supplies saved attention state for four shared text segments, totalling 416 tokens. Four follow-up requests, called children, each have a 512-token prompt and place those segments at different positions.

The shared-segment workload and the CacheBlend comparison

The TokenDance-style case supplies the multi-request workload and sharing information; CacheBlend supplies the selective recalculation. Original CacheBlend already addresses combining cached text chunks with different preceding context. Here the application knows which four children will reuse which segments. Our integration carries those exact segment identities and positions into each request and can coordinate the children as a group.

The experiment gives the cache process an unchanged source and restores it into each child's private model state. It then coordinates the children without letting one request alter another's working data. The prompts, shared segments and repair settings stay fixed so the comparison isolates loading and execution order.

A stock CacheBlend workload would also be a valid way to study DMA-BUF loading. This experiment measures our existing multi-request integration. It establishes no advantage of our additions over stock CacheBlend. A direct comparison would need a separate stock CacheBlend control with clearly defined segment inputs and recomputation settings, alongside full-prompt recalculation.

CacheBlend repair and the TokenDance-style integration

It uses our TokenDance-style recovery code built on LMCache's CacheBlend implementation. LMCache calls the feature CacheBlend, or cache blending. It implements the selective recalculation approach from the CacheBlend paper: reuse saved attention state and recalculate selected tokens to account for their changed preceding context.

In our fork, the one-request entry point is LMCBlender.blend(). Our TokenDance-style additions supply explicit shared text segments and their positions, give each request its own execution state, and force recalculation of every position without restored data. The four-request version uses our LMCBlender.blend_group() extension. This is our reproduction built with LMCache, rather than the TokenDance authors' implementation. The first comparison already includes our additions; it is not an unmodified upstream CacheBlend run.

First we run that extended serial implementation with loading inside the model process. Then we load the same saved segments through the separate cache process, place them in each child's private GPU memory, and run the same repair algorithm. Here serial means finishing one child before moving to the next. The shared source data stays unchanged; each child has its own working copy because repair changes model state.

Repair is needed because a token's saved state depends on the text before it, as well as its position. Adjusting key rotation fixes the position component. CacheBlend's selection determines which restored positions receive additional work, and our extension adds every position missing restored state. This is approximate reuse of earlier computation; the comparison is against the same method running serially, rather than recalculating the entire prompt.

Model outputs across the cache-process boundary

No differences were observed against that reference in this test. All four children have bit-for-bit identical saved keys and values and logits, the scores used to choose the next token. Their generated tokens and choices of which positions to recalculate also match. We check those choices at every layer: four children × 24 layers gives 96 comparisons.

Seven failure controls also pass, including missing or corrupt saved data and failure after GPU work has been submitted. Recalculation is allowed only after pending work finishes and releases the destination memory. If cleanup cannot complete, the request must wait or fail safely.

DMA-BUF loading measurements

We compare two ways of loading the saved segments from an NVMe SSD, a fast PCIe-connected solid-state drive. The model, data and repair algorithm stay the same. Children still run one at a time.

1. Host staging: SSD → locked host RAM → GPU memory. The RAM buffer is pinned so the device can access it throughout the transfer.
2. DMA-BUF: SSD → GPU memory through a registered buffer shared with the kernel. DMA-BUF is Linux's buffer-sharing interface; this route avoids the host staging step for the storage read.

This is a forced-read storage comparison. Singleflight is off; the retention fix is present and enabled in both routes. We run seven matched pairs, alternating which route goes first. Each pair runs all four children through each route, one child at a time.

Singleflight and shared-source reuse

The earlier engine singleflight mechanism joins requests with an identical prefix, so they can adopt one restored prefix inside vLLM. This workload puts shared text at different positions after different input. Each child therefore keeps its own position-adjusted and repaired model state. That exact-prefix mechanism is not the mechanism for sharing these interior segments. Prefix caching is disabled in this fixture so recovery is attributed to the explicitly named segments.

Sharing one stored source is still useful. Several children can use the same unchanged saved bytes while keeping their repaired GPU copies private. Retaining that source across children, and joining simultaneous loads of it, are the relevant storage optimizations. This particular transfer test runs children serially and forces a fresh load for every child, so it can time the SSD routes. The combined shifted-segment read-once serving test remains to be evaluated.

The test explicitly removes the four retained objects from LMCache's memory tier before every child. Each child therefore reads the same saved objects from SSD again: 12 MiB per child, 48 MiB across four children. This eviction makes every child exercise the selected SSD route instead of timing a hit in the memory cache. The repeated reads are deliberately requested by the benchmark, rather than caused by losing the retained copy unexpectedly.

The earlier read-once experiment had a different setup: singleflight was on, and the fixed retention kept one 48 MiB ancestor available across the children. It read that ancestor from SSD once. Here, forced eviction takes away that cross-child reuse to compare the two storage routes.

On a small screen, scroll the timing table sideways to see every column.

Negative values mean DMA-BUF took less time. For each pair we first take the median of its four children for each route, then compare those two medians. The table reports the median of those seven paired differences and ratios. The timed parts overlap: reads and corruption checking are inside restoration, and restoration is inside the final row. They must not be added together.

The storage read calls are faster in every pair, by a median 0.152 ms. That benefit does not consistently carry through to complete restoration, which is faster in four pairs and slower in three. A typical full restore takes about 17 ms per child; the outer interval with model repair and cleanup takes about 189 ms. These are instrumented replay timings, rather than measurements of how long a serving client waits for its first token.

The path still has costs beyond the read. To check for corruption, the GPU route copies the entire 12 MiB payload back to the CPU for a SHA-256 checksum. It also copies the shared GPU source into each child's private working memory. The table measures those remaining costs too. It does not demonstrate a path with no host copies.

GPU SHA-256: exact checksums, slower restoration

The tested GPU checksum removes the payload download but makes complete restoration slower in all seven matched pairs. We keep CPU hashing as the default for this profile. The result applies to our experimental CUDA kernel and the existing whole-object checksum format.

The original storage comparison above used CPU SHA-256 in both routes. This follow-up holds the DMA-BUF route fixed and changes only the checksum implementation. Both arms keep the retention fix enabled, singleflight off and explicit eviction before every child. Each child reads the same four padded 3 MiB objects from SSD, then uses the same CacheBlend repair code.

Before timing, every saved object receives an independent CPU digest. The GPU implementation must match it. During restoration, it hashes the same complete bytes, including padding, before position adjustment. Its four 32-byte digests return 128 bytes per child, replacing the checksum's 12 MiB payload download. This byte reduction is specific to the checksum; full model-state captures remain active.

For each pair, take the median of its four children for each arm. The table shows the medians across seven pairs and the median of their paired changes. Positive changes mean GPU hashing is slower. Read calls and checking are inside restoration, which is inside the final row; these intervals must not be added.

All model comparisons pass. The same-source reference and sixteen registered runs cover two poisoned-buffer arrival checks and fourteen timed arms. Each registered run matches the reference's raw keys, values, next-token scores, generated tokens, retrieval masks and all 96 per-layer recalculation choices. Maximum numerical differences are 0.0. Every arm completes its native reads and releases the source buffers with checked physical cleanup.

Four whole-object hashes limit parallel work

Our kernel assigns one GPU thread to each complete object. Standard SHA-256 processes each message's 64-byte blocks in order, carrying the previous result forward. A 3 MiB object has 49,152 data blocks, followed by the hash's padding block. The four objects supply only four independent chains.

A standalone test isolates this computation using deterministic synthetic bytes. Across fifteen trials with rotating method order, hashing the full 12 MiB set takes 5.512 ms with CPU hashing, including the download, 249.010 ms with four serial GPU calls, and 62.252 ms when one GPU call hashes all four objects concurrently. The concurrent case is an opportunity check; the model restore still hashes sources one at a time. Returning fewer bytes cannot compensate for this kernel's much longer computation.

This does not rule out a faster GPU implementation. It does show that moving our existing whole-object SHA-256 calculation onto this kernel is a poor optimization. Keep the measured CPU path while evaluating another candidate.

Further optimization targets

Overlapping independent SSD reads and GPU checks remains untested. It should first use a checksum implementation that can improve complete restoration. Changing to independently hashed smaller pieces could expose more parallel work, but it would change the integrity format; combining those digests does not reproduce the current whole-object SHA-256. That needs a separate design and matched comparison.

CUDA streams can overlap independent GPU work. Our io_uring reads are not automatically ordered in a CUDA stream: any new overlap must establish external I/O completion and GPU visibility before hashing, then retain buffers until both storage and GPU consumers finish.

This test is separate from the earlier read-once retention result. Here, forcing a fresh read for every child causes 48 MiB of reads across four children. In the earlier test, retention kept one 48 MiB ancestor available and removed repeat reads. The equal 48 MiB totals describe different workloads.

make defconfig-kmesh-registered-inputs-cpu
make kmesh-contracts-plan
# Set the pinned source, matching Python and external output paths, then run.
make kmesh-contracts
make defconfig-kmesh-registered-inputs-cuda
make kmesh-native-plan
make kmesh-native
make defconfig-kmesh-registered-model-cuda
make kmesh-registered-plan

# CPU ownership, host-RAM storage and buffer lifetime
make defconfig-kmesh-interior-spans-cpu
make kmesh-contracts-plan
KMESH_CONTRACTS_SOURCE=/path/to/pinned/LMCache \
  KMESH_CONTRACTS_VLLM_SOURCE=/path/to/pinned/vllm \
  KMESH_CONTRACTS_OUTPUT=/path/outside/checkouts/cpu-capture \
  KMESH_CONTRACTS_PY=/path/to/prepared/python make kmesh-contracts

# Separate-process native store/restore and raw release controls
make defconfig-kmesh-interior-spans-cuda
make kmesh-native-plan
KMESH_NATIVE_SOURCE=/path/to/pinned/LMCache \
  KMESH_NATIVE_OUTPUT=/path/outside/checkouts/cuda-capture \
  KMESH_NATIVE_PY=/path/to/prepared/python make kmesh-native

25. Testing layer-major execution across four related requests

We test a layer-major execution schedule for four related requests. Each child advances through one model layer before any child moves to the next. Bringing the children to the same model stage gives the runtime a place to coordinate work on their shared segments.

This experiment compares that schedule with finishing one child at a time. It checks model outputs and recovery after a failed child while each request keeps its own mutable state.

Layer-major execution

Label the four children A, B, C and D. Each must pass through the model's 24 layers. There are two execution orders:

“Layer-major” describes that loop order. It does not describe how data is arranged in memory, and it does not mean combining the four children's math into one batched calculation. This implementation still runs each child's tensor operations separately on one GPU worker.

Prior work and the scope of this experiment

The direct precedent is TokenDance's KV Collector, described by Zhuohang Bian and coauthors in 2026. Section 4.2 describes coordinating related requests at each model layer and batching their position adjustment and difference calculations. Section 5 describes a separate retrieval and computation generator for each request, advanced together across layers. TokenDance uses CacheBlend's selective recomputation as its default backend.

We apply this existing design rather than introduce a new KMesh scheduling algorithm. Our current test applies it to saved state supplied by a separate cache process, with private child buffers and checked failure cleanup. Tensor calculations remain separate. The test establishes correct outputs and cleanup; a speed or memory-capacity benefit remains unmeasured.

Shared sources and private request state

The group first loads all four segments for all four children: sixteen placements in total. The original saved segments are shared and remain unchanged. Each child keeps its own GPU destination blocks and working buffers, because its surrounding text and repair calculations can differ. The service checks every member before making the group's restored inputs available to the model.

The saved sources in this group test live in LMCache's host-RAM cache tier. Some implementation files call it “CPU-L1”; that name means a software cache in ordinary system RAM, not the processor's tiny hardware L1 cache. A separate cache process supplies the data to one model worker. The SSD comparison ran children one at a time. Combining that loading path with grouped execution remains untested.

Measured comparison

On October 24, the in-process serial reference, the serial path using the separate cache process and this group path all produced bit-for-bit identical keys, values and next-token scores. All four children's generated tokens and all 96 per-layer recalculation choices also match. Each child restores the same 416 shared tokens within a 512-token prompt.

Recovery after one child fails

We deliberately fail child B after it finishes its first actual model layer. A, C and D finish with the serial reference's generated tokens and recalculation choices. B waits for its pending work to finish, discards all restored state for its prompt, and retries by recalculating the whole prompt without reuse. Its generated tokens match the ordinary recalculation reference. All shared source handles close with no pending work.

This failure test checks safe fallback and sibling isolation. The mixed repaired/recalculated run checks generated tokens and cleanup; the successful group run is the one with the complete raw-state and next-token-score comparison.

make defconfig-kmesh-registered-groups-cpu
make kmesh-contracts-plan
# Set the pinned source, interpreter and external capture paths.
make kmesh-contracts
make defconfig-kmesh-registered-groups-cuda
make kmesh-native-plan
make kmesh-native
make defconfig-kmesh-registered-group-model-cuda
make kmesh-registered-plan