What does retaining shared context buy?

16. Does keeping data in LMCache stop repeat SSD reads?

Yes. We removed the repeated SSD reads and measured latency again. Section 15 prepares shared context in vLLM, but follow-up lookups can still reread that context from the SSD. Here we also keep a copy in LMCache, so those lookups find it in memory. Every measured group with retention reads the 48 MiB shared state from SSD exactly once, including preparation.

Is this section 15 retested with the read problem fixed?

It is a follow-up test of the fix, but it asks a narrower question. Both sides here prepare early, with a 250 ms head start. Both must finish warming before the follow-ups arrive. We compare keeping LMCache's copy against discarding that copy after loading vLLM.

Do not compare section 15's latency numbers directly with this table to claim the fix helped or hurt. These are different experiments. Section 20 makes the matched comparison with the read-saving behavior enabled in both approaches.

Host DRAM and DMA-BUF are different cache arrangements

External L1 means LMCache's in-memory cache outside vLLM. It is not the CPU's hardware L1 cache. Its memory can be host DRAM or a GPU buffer. The table tests retention off and on within each of these two paths:

The retained host arm is NVMe-backed host-DRAM caching; DRAM is a useful retention tier, not just a transfer buffer. Both arms start with empty resident caches and fetch from NVMe during preparation. A separate DRAM-only offload baseline would start with the state already in host RAM. This is therefore not a pure DMA-BUF on/off comparison with identical memory placement. Keeping both paths in one table makes their different resource costs and results visible.

Two stopwatches: A's first token, and everyone's eighth token

For a four-request test, we submit A, B, C and D together. Each uses the shared context and generates its own eight-token output. For an eight-request test, there are requests A through H. The two timing columns answer different questions:

1. How long does A wait for its first token? Start when A is submitted. Stop at A's first output token. This follows A, the first request submitted, even if B produces a token sooner.
2. How long until every request has generated 8 tokens? Start when we begin submitting the group. Stop when the last request finishes its eighth token. If D finishes last, the timer waits for D. It does not add the four requests' individual times.

The table would report 20 ms and 100 ms for that example. Neither number includes the earlier 250 ms preparation interval. The separate warm-up request produces one preparation token; it is not A, B, C or D and does not stop either timer. These short outputs test serving behavior, not the time needed to write a complete useful answer.

What changed after the repeated reads were removed?

These are the measured results. Each cell shows without retention → with retention. The SSD column counts all reads for preparation and the follow-up group. Both stopwatches start with follow-up submission, after preparation. Lower is better.

Fewer SSD reads do not make both stopwatches faster in every case. For example, with four requests on the GPU DMA-BUF path, A's first token arrives later: 71.071 → 82.974 ms. But the group reaches everyone's eighth token sooner: 181.381 → 170.288 ms. Both observations can be true. Across the four rows, A's first-token wait is mixed, while the median time until everyone has generated 8 tokens improves in every row. The published result data contains both measurements. There are five measured paired groups per case, after three warmups, with alternating order and one engine per path/group size. These are useful bounded observations, not a general latency or throughput guarantee.

For four follow-ups, the unretained path reads 240 MiB: 48 MiB for the warm request plus 48 MiB for each follow-up. Retention cuts this to one 48 MiB read for the entire group. Every follow-up reuses the already-loaded vLLM state; none needs another external load into vLLM.

The saving costs 48 MiB of additional occupied LMCache memory: host RAM in the staged path, GPU memory in the DMA-BUF path. The direct path therefore retains a second copy outside the one shared inside vLLM. Both controls configure a 512 MiB external arena and a 512 MiB engine KV pool; used external-cache bytes increase from zero to 48 MiB. This test applies no competing memory pressure.

Where would RAID flash fit?

A striped SSD array is a future backing-store test. It could prepare a large context sooner or serve more cold groups at once. In this retained 250 ms control, the context is already ready and the follow-ups trigger no SSD reads. Faster flash therefore cannot remove a remaining SSD wait from their measured interval; it would need a shorter head-start or cold-demand experiment to show that benefit.

Keep three experimental choices separate: where data starts (resident DRAM, one NVMe device or striped NVMe devices), how it reaches the engine (host staging or a qualified GPU DMA-BUF path), and when preparation starts (on demand or early). A DRAM-resident baseline remains valuable alongside the flash paths.

A software RAID test does not need a hardware RAID card or HBA. Linux can stripe reads across two NVMe devices in software. Ordinary reads into host RAM do not require GPU DMA-BUF support. Sending those reads directly into a GPU buffer is the separate question: the selected RAID stack, each drive's mapping and the PCIe route must support this experimental path. A working single-drive DMA-BUF test does not prove that RAID works too.

Two-drive RAID 0 is a useful optional test of read parallelism for reconstructible cache data; it provides no drive-failure redundancy. First check the topology and existing support. If qualifying the direct path needs new RAID or driver code, defer that work to a suitably equipped storage lab. The single-drive prefetch comparison in section 20 already demonstrates a benefit without RAID.

make defconfig-kmesh-retained-prefetch-cuda
make kmesh-restore-plan

# Set prepared model/source/interpreter paths as described in the README.
make kmesh-restore KMESH_RESTORE_PY=/path/to/prepared/python

KMESH_RESTORE_SECTION=retained KMESH_RESTORE_OUTPUT=/path/to/results \
  make kmesh-restore-verify
python tools/reproduce/kmesh_restore/publish_retained.py /path/to/results \
  --output public-results.json

17. Can the traces qualify family retention?

A family lease would protect vLLM's existing shared ancestor blocks between related requests. It would consume cache capacity that could serve other requests. The comparison must therefore count saved restoration or recomputation against displaced useful hits and unused holds. The external-L1 result above does not answer that engine-memory question.

The October 18 CPU audit of the public LMCache agent-trace corpus at its pinned revision finds 78 JSONL files, 1,915 unique parseable records, two exact duplicate records and 20 malformed nonempty lines. The malformed lines are in the MemGPT file. The Claude Code directory has capture tooling but no captured JSONL records. The source README labels PyCodeAGI as a synthetic pipeline.

None of the valid records supplies explicit parent links, invocation start and completion times, engine identity or physical cache events. There are zero observed invocation-lineage edges. Text-prefix overlap and spacing between record timestamps cannot establish fork gaps, simultaneous inference or eviction pressure. This audit therefore requires invocation capture before a trace-backed pressure comparison; it predicts no latency gain. The complete derived audit contains field coverage, counts and input hashes, without prompts or answers.

Reproduce the CPU audit

git clone https://github.com/LMCache/lmcache-agent-trace.git /path/to/traces
git -C /path/to/traces checkout 780bcc2979715150d8b9fd4737e026e625444cc9
make defconfig-kmesh-trace-audit
KMESH_TRACE_ROOT=/path/to/traces make kmesh-trace-plan
KMESH_TRACE_ROOT=/path/to/traces make
make kmesh-trace-test

Use Python 3.10 or later and a fresh output directory. No model or GPU is needed. The capture contract requires exact token IDs, explicit family and parent identities, terminal status and a shared clock domain. Even a complete invocation capture still needs physical block lifetimes, pool capacity, competing demand and eviction evidence. A controlled serving test can qualify a retention mechanism while leaving its production-workload benefit unresolved.

18. Does holding the ancestor pay under pressure?

The hold works, but it reduces total cached-token reuse in this controlled stress case. A bounded controller adds family references to vLLM's existing full ancestor blocks after the parent completes and before its request references are freed. It creates no second ancestor. The child later aliases those same physical blocks. Keeping them out of the eviction queue leaves less capacity for unrelated requests.

Measured on October 18, 2026, with Qwen2.5-0.5B-Instruct on a CUDA GPU, the experiment uses one synchronous eager engine, batch-invariant arithmetic, 16-token blocks and eight greedy output tokens per request. The pool has 129 physical blocks including one reserved null block. A 512-token ancestor occupies 32 blocks, or 25% of usable capacity. After the parent, the engine serves two passes over four unrelated 497-token prompts, then a child extending the parent's exact input and generated output with a 64-token tail. Each competing request needs 32 blocks with its continuation, so their working set fills the stock pool. This request order deliberately exposes capacity competition; it is not reconstructed from the public corpus.

Every one of the three alternating samples per arm has these counts. The hold saves 464 child-prefix tokens but displaces 1,856 competing cache-hit tokens, leaving 1,392 fewer cached tokens overall. The first competing pass is cold in both arms; the difference is in the second pass. These are cache-hit and eviction counts, not measured compute time, throughput or storage bytes. The samples share one engine and one fixed working set.

All 60 requests complete. All 240 paired generated tokens and their chosen log probabilities match exactly between stock and held arms. Each held child reuses the parent's exact 32 physical ancestor blocks with independent family and request references. All non-null references drain, and all 128 usable blocks return to the free queue. The capture records actual invocation boundaries, six exact parent-child edges, allocations, evictions and reference counts. The parent-to-child intervals include the competing requests, with no invented idle delay. Allocation observation remains active, so the durations are not a latency benchmark. The complete compact result retains every sample, runtime/source identities and original capture hashes.

Reproduce the controller and model capture

Use the prepared source, model and runtime instructions. The pinned vLLM serving source is unchanged. The controller and capture live in the reproduction harness; this is not a production scheduler integration.

export KMESH_RESIDENCY_SOURCE=/path/to/vllm
export OMP_NUM_THREADS=2 MKL_NUM_THREADS=2
make defconfig-kmesh-family-residency-cpu
KMESH_RESIDENCY_OUTPUT=/path/to/fresh-cpu-results \
  make KMESH_RESIDENCY_PY=/path/to/prepared/python

make defconfig-kmesh-family-residency-cuda
export KMESH_RESIDENCY_MODEL=/path/to/model-snapshot
export KMESH_RESIDENCY_OUTPUT=/path/to/fresh-serving-results
make kmesh-residency-plan
make KMESH_RESIDENCY_PY=/path/to/prepared/python
python -m tools.reproduce.kmesh_residency.publish /path/to/fresh-serving-results \
  --output public-results.json

CPU allocator checks cover overlapping families, bounded admission, generation-fenced release, expiry, cancellation and close. The model capture covers normal completion and physical aliasing. Expiry uses a 60-second deadline checked at invocation boundaries; caps on unique held blocks and active families bound residency and metadata. These limits are not a learned or trace-qualified selection policy.

The result supplies a concrete capacity-cost counterexample. It does not establish that all family retention loses, or that the hold is a useful default. No external storage connector, DMA-BUF route or cross-engine sharing is tested here. A production benefit still requires observed family timing and competing demand, with a policy that saves enough work to pay for displaced hits and unused holds. The trace gap in the preceding section remains open.

19. Does model-driven branching leave room for better retention?

Stock vLLM reaches the ideal exact-prefix cache ceiling in this unsuccessful local-model search. This capture lets a model generate arithmetic proposals, score candidate states and choose which states to extend. Later requests follow those choices. It answers whether stock prefix caching leaves exact-prefix reuse available to an ideal retention policy in this observed search, at the same physical cache capacity.

The workload uses the official Tree of Thoughts Game of 24 search at revision 8050e67d0e3a, with its original prompts, value memoization and dataset. We replace its model provider with Qwen2.5-7B-Instruct and take puzzles 0, 1 and 2 in file order. This bounded adaptation keeps two states, evaluates each once and allows 192 output tokens per call. It uses one synchronous eager vLLM engine, BF16, a 2,048-token context, 16-token cache blocks, temperature 0.7 and seed 123 plus the call index. All task outcomes count; this is not the paper's model or accuracy setup.

The October 20, 2026 (PDT) capture runs on a CUDA GPU with a configured 512 MiB cache and two CPU threads. The observed pool contains 585 blocks, including one reserved null block, leaving 584 usable blocks. The aggregate result records exact model, search, vLLM and runtime identities plus raw capture hashes. 173 model calls complete: 21 proposal calls and 152 value calls. Application memoization avoids 84 further value calls. The upstream driver treats 270 proposal lines as candidates and retains 24 across its steps. All failed tasks stay in the result.

Prefill means processing input tokens whose KV is not reused. The table counts those tokens, leaving output generation outside this metric.

The offline bound knows the complete future request order. It can retain full blocks independently of their prefix ancestors and ignores active-request pins. Generated full blocks enter as writes, and the final prompt fragment needed to produce next-token scores remains compulsory. These relaxations favor the ideal cache: the table gives an upper limit on possible token savings, not an implementable policy or a conversion into elapsed time. This closes further retention-policy comparisons for these captured calls at this budget. Successful reasoning workloads remain untested by this result.

A search parent is not automatically a reusable ancestor. All 170 recorded producer edges have known parents in one engine and clock domain. None preserves the parent's entire input and generated output as the child's exact prefix. The search prompts can rewrite a state for evaluation or extension, so a family label cannot justify sharing all of its KV. Application memoization avoids a call; candidate pruning does not cancel an inference request. This workload has one active call at a time and performs no live fork or cross-engine sharing.

All non-null references drain, and all 584 usable blocks return to the free queue; 1,290 cached blocks are evicted during the search. The complete invocation, search and physical allocator evidence passes consistency checks, and the CPU export regrades answers and recomputes the bound. Those checks do not independently replay tokenized search or authenticate execution. Source hashes and the admitted model run are checked separately. Capture observation and writes remain active, so durations are not latency results. No external cache connector or storage transfer participates here.

Reproduce the search and CPU evidence

Follow the pinned model, runtime and search preparation. The worker checks all eleven model/tokenizer hashes before allocation. Keep raw prompts, answers, token IDs and physical block identities with private evidence. The public exporter retains aggregate counts and capture hashes.

make defconfig-kmesh-tot-cuda
export KMESH_RESIDENCY_SOURCE=/path/to/vllm
export KMESH_RESIDENCY_WORKLOAD_SOURCE=/path/to/tot
export KMESH_RESIDENCY_MODEL=/path/to/model-snapshot
export KMESH_RESIDENCY_OUTPUT=/path/to/fresh-search-capture
export PYTHONPATH=/path/to/tot-deps:/path/to/vllm
export OMP_NUM_THREADS=2 MKL_NUM_THREADS=2
make kmesh-residency-plan
make KMESH_RESIDENCY_PY=/path/to/prepared/python

make kmesh-workload-test kmesh-headroom-test
make kmesh-headroom KMESH_HEADROOM_ROOT=/path/to/fresh-search-capture \
  KMESH_HEADROOM_OUTPUT=/path/to/fresh-headroom.json
python3 -m tools.reproduce.kmesh_residency.workload_publish \
  /path/to/fresh-search-capture --output public-workload-results.json

The profile has a 384-call limit, a 75-minute safety ceiling checked at call boundaries and a 120-second bound on each admitted invocation. Partial captures cannot qualify. This result concerns exact causal-prefix residency in these three sequential puzzles at this capacity. Approximate or position-independent recovery, cascade attention, storage latency and concurrent agent traffic need their own qualified tests. The remaining historical source work on multiprocess ownership and serial shifted restore continues independently of this retention decision.

20. Does loading shared context early actually help?

Why does this build on the earlier singleflight tests?

Singleflight was already enabled. External retention was the missing piece in the storage path. Singleflight lets related requests share one restored ancestor inside vLLM. In the earlier preloading tests, LMCache could still read that ancestor from SSD again when later lookups arrived. Keeping its external copy turns those lookups into memory hits.

This extends cascade's shared-prefix principle through offloading: the shared ancestor and private branches also guide restoration, external residency and advance preparation. The connection to cascade explains how these layers fit together. The combined performance benefit of cascade attention and these storage controls remains unmeasured.

The results build on each other: share one engine copy, retain the external copy, then prepare that state before the follow-ups arrive. The retention control directly measures the reduction in repeated SSD work. The larger comparison below keeps that read-once behavior in both controls and measures the additional benefit of starting preparation early. The older control has five measured pairs per case; the new comparison has 200 measured groups per control in each path and group size.

What are we comparing?

Imagine four requests, A, B, C and D, that all need the same saved context. The context is 48 MiB of cached model state on an SSD. Each request adds its own input and asks the model to generate eight output tokens. A token is a piece of text, sometimes part of a word; eight tokens need not form a complete answer. We also test groups of eight requests.

• Load on arrival: wait for the requests, then load their shared context once and let them reuse it.
• Load early: begin preparing that same context 100 ms before the requests are due. Load it into vLLM and run a small, one-token warm-up request, so the follow-ups arrive with their shared state ready.

Both approaches already read the SSD data only once. The benefit comes from doing preparation before the requests arrive. It is not a saving from eliminating extra SSD reads in only one side of the comparison. This is the comparison that sections 15 and 16 left open.

A concrete result: four requests using DMA-BUF

Start both stopwatches when the group is due to arrive. A is the first request submitted; it is not necessarily the first request to produce output.

A starts producing text much sooner, and the group finishes a little sooner. Those are two different benefits. “About half the wait” describes A's first token; it does not mean every request improves by half or the whole job runs twice as fast. Values are means; savings are calculated before rounding.

Does the benefit hold across the tested cases?

Yes. All four combinations improve on both measures. Host staging moves SSD data through CPU RAM on its way to vLLM. DMA-BUF moves it into a separate GPU buffer, then copies it into vLLM's cache.

Each row compares 200 groups per approach, spread across ten fresh paired engine runs. The statistical intervals support improvements on both measures in every row. This is stronger evidence for this controlled workload than the small comparisons in sections 15 and 16. Full times and uncertainty intervals are available below.

The 100 ms is advance notice, not free work

The test tells the system when the requests will arrive. It starts early preparation 100 ms before that time. It does not postpone the requests until preparation finishes: any delay in submitting them still counts against the measured wait. The preparation includes loading, copying into vLLM and one real warm-up token. This measures the complete preparation method, not just SSD transfer speed.

An application needs an opportunity to do that work early. For example, it might know which context a follow-up will need while waiting for a tool. This benchmark supplies that knowledge; it does not demonstrate an application predicting it. Delaying useful requests to create a 100 ms gap would have to count as extra waiting.

The pilot tried 0, 25, 100 and 250 ms of notice. At 25 ms, preparation was not ready in any of its ten groups on either path. At 100 ms it was ready in all ten, so 100 ms was selected for the larger test. This is the shortest successful tested lead, not a measured minimum or the SSD's loading time. All 800 measured early-loading groups in the larger test were ready by arrival.

The tables count waiting after the scheduled arrival. The benchmark also records an earlier, fixed 250 ms interval before arrival. For the four-request DMA-BUF example, counting from that earlier start gives 367.89 ms versus 354.06 ms: the same roughly 13.84 ms saved, but about 3.8% of the longer interval. Neither percentage is a prediction for an entire real application workflow.

Why can early SSD loading look faster than data already in DRAM?

“Already in DRAM” means the saved state is in LMCache's CPU memory. It still has to be restored into vLLM's GPU cache after the requests arrive. Early loading from SSD has already done that restoration and the warm-up before arrival. These controls differ in how much work remains; they do not show that SSD is faster than RAM.

The “engine already resident” control starts with the state in vLLM's cache. It is the closer comparison for a fully prepared request. In the small four-request pilot, early loading and that control give similar first-token waits. The placement table below keeps the actual DRAM and GPU memory controls visible.

What this result establishes

KMesh can use advance notice to reduce waiting even after duplicate reads have been removed. The next question is how often a real application can supply useful notice, and whether the benefit survives competition for storage and GPU memory. A wrong hint still spends I/O, memory and model work without helping a follow-up.

The evidence here covers one small model, a 48 MiB shared context, four or eight requests, and eight-token outputs, with no competing families or memory pressure. It supports early preparation in that setting. It does not establish a production prediction policy, a general DMA-BUF latency advantage, or a combined speedup with Cascade attention.

make defconfig-kmesh-prefetch-cuda
export KMESH_RESTORE_PY=/path/to/prepared/python
export KMESH_RESTORE_MODEL_DIR=/path/to/model-snapshot
export KMESH_RESTORE_VLLM_SOURCE=/path/to/vllm
export KMESH_RESTORE_LMCACHE_SOURCE=/path/to/LMCache
export PYTHONPATH=/path/to/LMCache:/path/to/vllm
make kmesh-restore-plan
make kmesh-restore
KMESH_RESTORE_PHASE=pilot \
  KMESH_RESTORE_OUTPUT=results/kmesh-prefetch/pilot make kmesh-restore
KMESH_RESTORE_PHASE=confirmation \
  KMESH_RESTORE_OUTPUT=results/kmesh-prefetch/confirmation make kmesh-restore