KV-cache sharing · a state-of-the-art tracker · October 2026
Can another model pick up where the first one left off?
A small model has already read a long conversation. A larger model takes over. Why should it have to read the whole thing again? Cross-model KV-cache sharing tries to pass on the work, not just the text. The catch is that each model stores that work in its own representation.
We track the papers trying to cross that boundary and test their ideas in knlp. This page follows the promise through to the result: what worked, what broke, and which questions are still worth asking.
Where the evidence standsSharing can work between closely related models: DroidSpeak preserved answer quality for one Mistral adapter pair after rebuilding just 4 of 32 layers. The broader translation recipes we tested still lost too much quality. RaReCache's choice of what to rebuild helped, and modified MoT's translator added useful information, but neither result made its tested recipe a substitute for the receiver's own cache.
KV-Lingo · translateDroidSpeak · rebuild layersMoT · translate + replayRaReCache · choose what to rebuild
Evidence through October 19, 2026. These are bounded retests and independent implementations, not a ranking of every paper or a matched leaderboard.
Four ways to avoid reading the same context twice
When a model reads a prompt, it builds attention keys and values for each token. That first pass is called prefill; the saved keys and values are its KV cache. The model taking over is the receiver. Its own prefill gives us the reference answer quality. Sharing is useful only if it saves enough work without losing too much of that quality.
Translate the missing history
Each model keeps its own cache. When it returns to the conversation, learned key and value maps translate the spans it missed.
Our test: Qwen3-4B and Qwen3-8B alternate for ten turns on 32 CoQA conversations.
Did not pass in both starting orders
The KV-Lingo result →
Rebuild the layers that matter
The receiver reuses most of the sender's cache and recomputes one contiguous layer interval, chosen on separate profiling data.
Our test: one handoff between two Mistral-7B adapters, with 50 profiling and 50 held-out HotpotQA records.
Preserved quality for this pair
The DroidSpeak result →
Translate into a smaller replay
Source attention chooses at most 128 prior keys. The receiver processes the context with those selected keys, and learned translators supply K/V at selected layers.
Our test: GPT-2 to GPT-2-medium, 500 items across five tasks, with four controls for separating the effects.
Translator helped; recipe fell short
What the controls revealed →
Choose what to rebuild
Map the sender's cache to the receiver, then rank the parts that need native receiver repair. Compare that choice with a simple, equal-count repair policy.
Our test: Qwen3-0.6B to Qwen3-14B on all 1,319 GSM8K test questions, including fixed 30% and 50% repair.
Better selection; quality margin missed
Where repair helped →
What counts as a useful result?
A paper can establish a promising idea without settling whether it works for the models and workloads we want to serve. We keep three questions separate when we retest it.
Does the mechanism work?
Check the implementation first
Token positions, layer alignment, cache ownership, and replay must be correct. The exact-cache controls should reproduce the receiver before a learned map gets credit or blame.
Does it keep the answer?
Compare with the receiver
Measure the candidate against a matched native run and simpler alternatives. A gain over a weak translation baseline can be real while still falling short of usable quality.
Does it save work overall?
Count the whole handoff
Include sender work, transfer, translation, recomputation, and retained cache copies. A faster local kernel or fewer rebuilt layers does not establish a cheaper serving system.
How to read the paper names: KV-Lingo is an independent implementation motivated by the paper. DroidSpeak is a bounded mechanism retest with an identifiable paper model pair. RaReCache is an independent reconstruction with recorded recipe differences. Modified MoT includes changes to the author implementation. Their results apply to those tested versions; they do not reproduce every headline claim in the original papers.
What each method asks of the receiver
These three implementations make different trade-offs in training, retained state, and receiver work. RaReCache's repair-budget comparison is reported separately below.
Implementation details for the translation, layer-repair, and replay paths.
| Axis | KV-Lingo | DroidSpeak | Modified MoT |
| Repair operation | Translate keys and values with learned, layer-specific linear maps. | Recompute one native receiver-layer interval; reuse sender KV elsewhere. | Select prior keys from source attention, replay the receiver, and inject translated K/V at target layers. |
| New training | Yes. Each directed model pair needs fitted key and value maps. | No new weights. An offline profiler selects an interval for the model pair. | Yes. Two translator experts use Top-1 routing for the fixed directed model pair. |
| Receiver compute | Apply maps to missing spans, then continue normal decoding. | Run the selected receiver layers over the cached prompt, then continue decoding. | Run exact bottom layers and a complete selected-key receiver replay; inject at layers 17, 19, 21, and 23. |
| Compatibility | Needs defined layer correspondence and trained maps. It can bridge different hidden widths when the maps have matching source and target shapes. | Our implementation requires the same decoder architecture, layer layout, hidden shape, and token vocabulary. | The tested GPT-2 pair shares token IDs but has different widths and depths. The frozen recipe uses explicit depth-ratio channel alignment. |
| Cache ownership | One retained cache line per participating model, with translated copies of missing spans. | One hybrid cache: sender KV outside the interval and receiver KV inside it. | A source cache and selection ledger feed a receiver replay with optional translated injections. |
| Repeated switches | Central to the tested ten-turn protocol. | Our current result tests a single sender-to-receiver handoff. | Not tested. The attribution run evaluates one frozen source-to-receiver path per item. |
| Measured quality result | No checkpoint pair passed the full either-model-first rule. Three of four passed both controls only when 8B started; that is unconfirmed. | The four-layer interval passed the held-out receiver-relative quality rule. Direct full reuse failed. | Translation improved NewsQA over the same replay without injection by 2.68 F1 points, but remained 8.56 points below native. Other task intervals crossed zero. |
| Measured system result | No end-to-end latency, cost, or memory saving established. | A 4.03× local time-to-first-token ratio is a compute-path signal, not a distributed-system speedup. | No serving latency or memory saving established. Four diagnostic generation arms are not a serving benchmark. |
| Current status | Closed for this unrestricted Qwen3-4B/8B recipe. The restricted 8B-first observation has not been confirmed on untouched data. | Passed the quality rule for one adapter pair. A complete remote serving test remains open. | Translation helped on NewsQA. The fixed recipe still missed the quality requirement, so we stopped it. |
The scores answer different questions. KV-Lingo used CoQA and ten alternating turns. DroidSpeak used HotpotQA and a profile/held-out split. Modified MoT used five 100-item cohorts and a different GPT-2 pair. The models, tasks, controls, and quality rules differ, so a larger score here does not make one method the winner.
What survived the retests
The encouraging result is that some cached work really can cross a model boundary. The harder question is how much repair it needs. Token F1 below measures normalized word overlap with the reference answer; each result is compared with its own matched controls.
KV-Lingo outcome
0 / 4
None of four forward/reverse checkpoint pairs passed the full quality rule with either model starting and against both native references.
All eight 4B-start comparisons failed at least one requirement. Starting with 8B looked better, but that narrower observation still needs fresh confirmation.
DroidSpeak outcome
4 / 32 layers
The selected receiver interval scored 0.3149 held-out F1 versus 0.2887 for full receiver prefill and a required floor of 0.2742.
Direct cache reuse scored 0.2263. Some receiver recomputation was necessary.
Modified MoT outcome
+2.68 F1
On NewsQA, translated selected replay scored 11.11% versus 8.43% without injection. The paired 95% interval was +0.52 to +4.82 points.
The receiver's own cache scored 19.67%. Translation recovered about 24% of the gap left by replay, which is useful evidence but still a long way from preserving the reference quality.
A real openingDroidSpeak preserved quality for the tested pair of adapters on the same base model. Modified MoT also shows that a learned translation can add information the reduced replay would otherwise lose.
The missing proofWe have not demonstrated a general translator across model families, reliable repeated handoffs for the failed recipes, or an end-to-end serving saving. Those remain separate tests.
What you can inspectThe public code and source pins are linked below. The exact tested KV-Lingo and modified-MoT checkpoint tensors are not published. DroidSpeak needs a chosen layer interval rather than newly trained weights.
MoT: did translation fail, or did replay lose the context?
A weak final answer does not tell us which part of a pipeline failed. We kept the translator and key selection fixed, then changed only what the receiver was given. This GPT-2 to GPT-2-medium test used five 100-item task cohorts, at most 128 selected keys (Top-S=128), two exact bottom layers, greedy decoding, and a 64-token output cap. Every item ran all four versions:
N · native
Exact receiver cache
The receiver reads the full context itself. This is the quality reference.
T · translated
Selected replay + learned K/V
The candidate: replay the context with fewer keys and inject the frozen translator's K/V.
R · replay
Selected replay, no injection
Run the same selected-key replay without translated K/V. Comparing T with R tells us what the translator adds.
O · native injection
Selected replay + native K/V
Inject the receiver's full-prefill K/V at the same layers. This tests whether that cache helps inside reduced replay; it is not an upper bound.
Scores from the frozen four-way test, in percentage points. The interval compares translated replay (T) with replay alone (R).
| Task | N | T | R | O | T−R, 95% interval |
| BoolQ accuracy | 56.00 | 61.00 | 59.00 | 62.00 | +2.00 [−5.00, +9.00] |
| MMLU-Redux accuracy | 17.00 | 16.00 | 17.00 | 17.00 | −1.00 [−6.12, +4.00] |
| NewsQA token F1 | 19.67 | 11.11 | 8.43 | 8.90 | +2.68 [+0.52, +4.82] |
| PubMedQA accuracy | 29.00 | 37.00 | 41.00 | 36.00 | −4.00 [−10.00, +2.00] |
| SQuAD token F1 | 38.63 | 41.40 | 38.50 | 37.97 | +2.90 [−1.72, +8.74] |
The translator contributedEvery NewsQA context exceeded the 128-key budget. Adding translated K/V improved F1 by 2.68 points; the interval, grouped by document, excludes zero. The learned map did add useful information.
Most of the gap remainedReplay alone lost 11.24 points against native on NewsQA. Translation recovered about 24% of that loss and still finished 8.56 points behind. Improving the translator alone would not explain away the larger replay deficit.
Keeping every key was exactOn all 189 contexts that fit within 128 keys, replay without injection produced exactly the native tokens. Differences appeared when the method actually removed keys.
What this leaves open: selecting the wrong keys and changing how later layers attend to them are still entangled. Even exact K/V from full prefill may not fit the state created by reduced replay, which is why O is not a ceiling. The layer-23 injection has no downstream prefix-cache layer; only the first three sites can change the returned cache. Other T−R intervals cross zero, SQuAD has only four context clusters, and several classification baselines are weak or produce invalid answers. We stopped this recipe here. This experiment measured answer quality, not serving speed or memory savings.
RaReCache: choosing the repair helps, but how much is enough?
Our independent reconstruction tested Qwen3-0.6B to Qwen3-14B on all 1,319 GSM8K math questions. Repairing more of the receiver's cache raised accuracy. Choosing what to repair by rank also beat a uniform choice with the same repair count. Both are useful results. The remaining question was whether either choice came close enough to the receiver's own cache.
Fixed repair budgets on the same GSM8K test set. Accuracy uses the frozen explicit-final scorer.
| Path | Correct / 1,319 | Accuracy | Meaning |
| Native Qwen3-14B | 1,253 | 94.9962% | The matched receiver reference. |
| Rank-selected 30% repair | 1,140 | 86.4291% | The lower repair budget. |
| Rank-selected 50% repair | 1,206 | 91.4329% | More repair helps, but the quality requirement still fails. |
| Uniform 50% repair | 1,181 | 89.5375% | Same repair count, simpler selection. |
At 50% repair, rank selection gained 1.8954 percentage points over uniform selection, with a paired 95% interval of +0.2274 to +3.5633. It still trailed native by 3.5633 points. The predeclared rule required the one-sided 95% upper bound on that loss to stay within 3 points; the measured bound was 4.6247. Both 50% variants missed the quality rule.
A partial result, not a near-native reproduction. The paper reports 95.3% at 50% repair and 95.1% for native. Our native anchor is close; our repaired result is lower. Differences include prompting, calibration, source-layer selection and sharing, and scoring. We have not isolated which difference explains the gap. The fixed configuration is closed, and these diagnostic runtimes do not establish a serving speedup.
Can native recomputation rescue a translated cache?
DroidSpeak makes this an attractive idea, but the repair location and model relationship matter. Its positive result used two adapters on the same base model. Our cross-width Qwen3 test rebuilt the receiver's first layers and used frozen KV-Lingo maps above them. That is a different experiment.
The tested prefix repair fell shortRebuilding the first 2, 4, 8, or 12 layers did not pass the quality rules in both directions on the 32-conversation development screen. The 4B→8B deficits were 13.07–16.35 F1 points against a 3-point limit. In reverse, the best gain over translation alone was just 0.52 points.
No boundary was selected, so timing, repeated handoffs, and fresh confirmation did not run. More depth search with these frozen maps is not the next step.
A middle-layer bridge is still a hypothesisRebuilding a middle interval would need a compatible hidden state at its entrance. Different-width models need an explicit bridge for that state, as well as layer alignment. The prefix test did not evaluate such a bridge, and the DroidSpeak adapter result does not prove it will work.
A new test should show what this added machinery buys over the receiver's exact cache and simpler repair controls before scaling up.
The prefix-repair figures summarize an independent experiment. Its exact run source and raw records are not published; this section is not an exact-replay package.
Where LMCache fits
Once a cache is compatible, it still needs to reach the process that will use it. LMCache handles that storage and movement. Our same-model control retrieved 7,168 cached tokens and produced an identical completion. A handoff to a different model adds another job: making the incoming state usable by that receiver.
Transport
LMCache
Names, stores, moves, and retrieves compatible KV data across processes or nodes.
Representation repair
KV-Lingo
Maps cached keys and values from one model representation to another.
Representation repair
DroidSpeak
Reuses compatible sender layers and rebuilds one interval with receiver compute.
Representation + replay
Modified MoT
Stores source state and a selection ledger, then runs receiver replay with translated injections.
The practical combination: use a cache service to carry the tensors, and a qualified receiver path to translate or rebuild them. The serving engine must check the model pair, token positions, selected keys, and replay recipe before reuse. Keeping an exact cache for each receiver remains the quality reference when comparing storage and latency costs.
Can we learn the next lesson without a large GPU run?
That is the next research question. Prefix Integrity Analysis already checks cache identity and block-position contracts on a CPU without loading a model. Those manifest checks do not establish that real cache tensors reload correctly or preserve an answer. We want to add a second cheap screen: small, trained models whose saved checkpoints let us test whether sharing preserves an answer.
Catch broken cache contracts. Reuse the existing CPU checks for identity, geometry, and query-dependent state. A cache that silently changes meaning between requests has failed before answer quality enters the picture.
Test with capable small models. Train compact models on controlled retrieval and conversation tasks, freeze their checkpoints, and compare native caches, translation, replay, and repair. Include both handoff directions and repeated switches.
Check whether the screen predicts anything. Compare its outcomes with known larger-model successes and failures, then test new methods on untouched cases. Only a calibrated screen should influence which expensive reproductions we run next.
Proposed, not yet validated. Small models can represent attention layouts and cache mechanics without representing all the behavior of a large language model. A toy success is not a paper reproduction, and a toy failure is not a refutation. The value of this screen depends on demonstrating which findings transfer across scale.
Try the modified-MoT implementation
The code is available if you want to inspect the mechanism or run your own comparison. This is the modified path discussed above, not a claim to reproduce the original paper. Use Python 3.10 or 3.11 with the pinned MoT requirements, and keep model snapshots and generated artifacts outside the source trees. The MoT fork retains the upstream PolyForm Noncommercial License.
Pin both repositories. The modified source commit is public on a dedicated branch. The knlp wrapper checks the exact seven-commit chain and key file hashes.
Run code and mechanics controls. Unit tests need no model. Real-model controls require the pinned GPT-2 and GPT-2-medium snapshots recorded in recipe.py.
Train and evaluate explicitly. Freeze the data manifest, run the diagnostic controls, train one directed checkpoint, then run the paired evaluation and four-arm attribution.
git clone https://github.com/mcgrof/knlp.git
git clone https://github.com/mcgrof/MoT.git modified-MoT
git -C modified-MoT checkout 9fb5e6f2a63981a53da1769c72e000a840ac976b
python3.11 -m venv mot-env
mot-env/bin/python -m pip install -r modified-MoT/requirements.txt pytest zstandard
cd knlp
../mot-env/bin/python -m pytest -q tests/kv_translate/test_mot_author.py
make defconfig-mot-author-readiness
make MOT_SOURCE=../modified-MoT MOT_AUTHOR_PYTHON=../mot-env/bin/python
The real-model mechanics gate checks native continuation, full-key replay, selected-key replay, token identity, and source provenance before training:
make defconfig-mot-author-mechanics
make mot-author-mechanics-preflight \
MOT_VARIANT=modified-mot \
MOT_SOURCE=../modified-MoT \
MOT_AUTHOR_PYTHON=../mot-env/bin/python \
MOT_SOURCE_MODEL=/path/to/gpt2-607a30d783dfa663caf39e06633721c8d4cfcd7e \
MOT_TARGET_MODEL=/path/to/gpt2-medium-6dcaa7a952f72f9298047fd5137cd6e4f05f41da \
MOT_OUTPUT=/path/to/new-run/mechanics
After you create a frozen manifest and checkpoint with the documented diagnostic targets, run the same four-arm evaluator:
make defconfig-mot-modified-attribution
make mot-author-attribution-eval \
MOT_SOURCE=../modified-MoT \
MOT_CHECKPOINT=/path/to/checkpoint \
MOT_CHECKPOINT_SHA256=<translator-file-sha256> \
MOT_DATA_MANIFEST=/path/to/diagnostic-data.json \
MOT_ANCHOR_RECORDS=/path/to/paired-records.jsonl.zst \
MOT_OUTPUT=/path/to/new-run/attribution
Available in the repositoriesThe exact source chain, configuration, data-free controls, model-revision checks, training path, evaluator, offline replay, and aggregate calculation are public.
Needed for an exact replayThe tested translator checkpoint, frozen 500-item manifest, and token-level anchor records are not in the public repository. You can reproduce the procedure with newly generated inputs, but not byte-replay this exact result without those artifacts.
Start with the full runbook: research/kv_translate/mot_author/README.md. It includes data freezing, training smoke, full training, paired evaluation, and offline replay commands.
Follow the evidence
KV-Lingo resultTraining, multi-turn protocol, all 16 comparisons, paper differences, and limits.
DroidSpeak resultPartial recomputation, profile and held-out scores, timing boundary, and LMCache control.
Modified MoT integrationDefconfigs, controls, training wrapper, attribution evaluator, tests, and runbook.
Modified MoT sourcePublic seven-commit source chain ending at the exact pinned attribution commit.
Prefix Integrity AnalysisCheap cache-contract checks, and why cache safety and answer quality need separate tests.
Research implementationsCode and tests for following up the questions on this page.
This page contains static aggregate values. It does not load private data or machine-readable experiment artifacts.