KV-cache sharing · a state-of-the-art tracker · October 2026

Can another model pick up where the first one left off?

A small model has already read a long conversation. A larger model takes over. Why should it have to read the whole thing again? Cross-model KV-cache sharing tries to pass on the work, not just the text. The catch is that each model stores that work in its own representation.

We track the papers trying to cross that boundary and test their ideas in knlp. This page follows the promise through to the result: what worked, what broke, and which questions are still worth asking.

Where the evidence standsSharing can work between closely related models: DroidSpeak preserved answer quality for one Mistral adapter pair after rebuilding just 4 of 32 layers. The broader translation recipes we tested still lost too much quality. RaReCache's choice of what to rebuild helped, and modified MoT's translator added useful information, but neither result made its tested recipe a substitute for the receiver's own cache.
KV-Lingo · translateDroidSpeak · rebuild layersMoT · translate + replayRaReCache · choose what to rebuild

Evidence through October 19, 2026. These are bounded retests and independent implementations, not a ranking of every paper or a matched leaderboard.

Four ways to avoid reading the same context twice

When a model reads a prompt, it builds attention keys and values for each token. That first pass is called prefill; the saved keys and values are its KV cache. The model taking over is the receiver. Its own prefill gives us the reference answer quality. Sharing is useful only if it saves enough work without losing too much of that quality.

Translate the missing history

Each model keeps its own cache. When it returns to the conversation, learned key and value maps translate the spans it missed.

Our test: Qwen3-4B and Qwen3-8B alternate for ten turns on 32 CoQA conversations.

Did not pass in both starting orders

The KV-Lingo result →

Rebuild the layers that matter

The receiver reuses most of the sender's cache and recomputes one contiguous layer interval, chosen on separate profiling data.

Our test: one handoff between two Mistral-7B adapters, with 50 profiling and 50 held-out HotpotQA records.

Preserved quality for this pair

The DroidSpeak result →

MoT paper · modified here

Translate into a smaller replay

Source attention chooses at most 128 prior keys. The receiver processes the context with those selected keys, and learned translators supply K/V at selected layers.

Our test: GPT-2 to GPT-2-medium, 500 items across five tasks, with four controls for separating the effects.

Translator helped; recipe fell short

What the controls revealed →

RaReCache paper · reconstructed here

Choose what to rebuild

Map the sender's cache to the receiver, then rank the parts that need native receiver repair. Compare that choice with a simple, equal-count repair policy.

Our test: Qwen3-0.6B to Qwen3-14B on all 1,319 GSM8K test questions, including fixed 30% and 50% repair.

Better selection; quality margin missed

Where repair helped →

What counts as a useful result?

A paper can establish a promising idea without settling whether it works for the models and workloads we want to serve. We keep three questions separate when we retest it.

Does the mechanism work?

Check the implementation first

Token positions, layer alignment, cache ownership, and replay must be correct. The exact-cache controls should reproduce the receiver before a learned map gets credit or blame.

Does it keep the answer?

Compare with the receiver

Measure the candidate against a matched native run and simpler alternatives. A gain over a weak translation baseline can be real while still falling short of usable quality.

Does it save work overall?

Count the whole handoff

Include sender work, transfer, translation, recomputation, and retained cache copies. A faster local kernel or fewer rebuilt layers does not establish a cheaper serving system.

How to read the paper names: KV-Lingo is an independent implementation motivated by the paper. DroidSpeak is a bounded mechanism retest with an identifiable paper model pair. RaReCache is an independent reconstruction with recorded recipe differences. Modified MoT includes changes to the author implementation. Their results apply to those tested versions; they do not reproduce every headline claim in the original papers.

What each method asks of the receiver

These three implementations make different trade-offs in training, retained state, and receiver work. RaReCache's repair-budget comparison is reported separately below.

Implementation details for the translation, layer-repair, and replay paths.
AxisKV-LingoDroidSpeakModified MoT
Repair operationTranslate keys and values with learned, layer-specific linear maps.Recompute one native receiver-layer interval; reuse sender KV elsewhere.Select prior keys from source attention, replay the receiver, and inject translated K/V at target layers.
New trainingYes. Each directed model pair needs fitted key and value maps.No new weights. An offline profiler selects an interval for the model pair.Yes. Two translator experts use Top-1 routing for the fixed directed model pair.
Receiver computeApply maps to missing spans, then continue normal decoding.Run the selected receiver layers over the cached prompt, then continue decoding.Run exact bottom layers and a complete selected-key receiver replay; inject at layers 17, 19, 21, and 23.
CompatibilityNeeds defined layer correspondence and trained maps. It can bridge different hidden widths when the maps have matching source and target shapes.Our implementation requires the same decoder architecture, layer layout, hidden shape, and token vocabulary.The tested GPT-2 pair shares token IDs but has different widths and depths. The frozen recipe uses explicit depth-ratio channel alignment.
Cache ownershipOne retained cache line per participating model, with translated copies of missing spans.One hybrid cache: sender KV outside the interval and receiver KV inside it.A source cache and selection ledger feed a receiver replay with optional translated injections.
Repeated switchesCentral to the tested ten-turn protocol.Our current result tests a single sender-to-receiver handoff.Not tested. The attribution run evaluates one frozen source-to-receiver path per item.
Measured quality resultNo checkpoint pair passed the full either-model-first rule. Three of four passed both controls only when 8B started; that is unconfirmed.The four-layer interval passed the held-out receiver-relative quality rule. Direct full reuse failed.Translation improved NewsQA over the same replay without injection by 2.68 F1 points, but remained 8.56 points below native. Other task intervals crossed zero.
Measured system resultNo end-to-end latency, cost, or memory saving established.A 4.03× local time-to-first-token ratio is a compute-path signal, not a distributed-system speedup.No serving latency or memory saving established. Four diagnostic generation arms are not a serving benchmark.
Current statusClosed for this unrestricted Qwen3-4B/8B recipe. The restricted 8B-first observation has not been confirmed on untouched data.Passed the quality rule for one adapter pair. A complete remote serving test remains open.Translation helped on NewsQA. The fixed recipe still missed the quality requirement, so we stopped it.

The scores answer different questions. KV-Lingo used CoQA and ten alternating turns. DroidSpeak used HotpotQA and a profile/held-out split. Modified MoT used five 100-item cohorts and a different GPT-2 pair. The models, tasks, controls, and quality rules differ, so a larger score here does not make one method the winner.

What survived the retests

The encouraging result is that some cached work really can cross a model boundary. The harder question is how much repair it needs. Token F1 below measures normalized word overlap with the reference answer; each result is compared with its own matched controls.

KV-Lingo outcome
0 / 4

None of four forward/reverse checkpoint pairs passed the full quality rule with either model starting and against both native references.

All eight 4B-start comparisons failed at least one requirement. Starting with 8B looked better, but that narrower observation still needs fresh confirmation.

DroidSpeak outcome
4 / 32 layers

The selected receiver interval scored 0.3149 held-out F1 versus 0.2887 for full receiver prefill and a required floor of 0.2742.

Direct cache reuse scored 0.2263. Some receiver recomputation was necessary.

Modified MoT outcome
+2.68 F1

On NewsQA, translated selected replay scored 11.11% versus 8.43% without injection. The paired 95% interval was +0.52 to +4.82 points.

The receiver's own cache scored 19.67%. Translation recovered about 24% of the gap left by replay, which is useful evidence but still a long way from preserving the reference quality.

A real opening

DroidSpeak preserved quality for the tested pair of adapters on the same base model. Modified MoT also shows that a learned translation can add information the reduced replay would otherwise lose.

The missing proof

We have not demonstrated a general translator across model families, reliable repeated handoffs for the failed recipes, or an end-to-end serving saving. Those remain separate tests.

What you can inspect

The public code and source pins are linked below. The exact tested KV-Lingo and modified-MoT checkpoint tensors are not published. DroidSpeak needs a chosen layer interval rather than newly trained weights.

MoT: did translation fail, or did replay lose the context?

A weak final answer does not tell us which part of a pipeline failed. We kept the translator and key selection fixed, then changed only what the receiver was given. This GPT-2 to GPT-2-medium test used five 100-item task cohorts, at most 128 selected keys (Top-S=128), two exact bottom layers, greedy decoding, and a 64-token output cap. Every item ran all four versions:

N · native

Exact receiver cache

The receiver reads the full context itself. This is the quality reference.

T · translated

Selected replay + learned K/V

The candidate: replay the context with fewer keys and inject the frozen translator's K/V.

R · replay

Selected replay, no injection

Run the same selected-key replay without translated K/V. Comparing T with R tells us what the translator adds.

O · native injection

Selected replay + native K/V

Inject the receiver's full-prefill K/V at the same layers. This tests whether that cache helps inside reduced replay; it is not an upper bound.

Scores from the frozen four-way test, in percentage points. The interval compares translated replay (T) with replay alone (R).
TaskNTROT−R, 95% interval
BoolQ accuracy56.0061.0059.0062.00+2.00 [−5.00, +9.00]
MMLU-Redux accuracy17.0016.0017.0017.00−1.00 [−6.12, +4.00]
NewsQA token F119.6711.118.438.90+2.68 [+0.52, +4.82]
PubMedQA accuracy29.0037.0041.0036.00−4.00 [−10.00, +2.00]
SQuAD token F138.6341.4038.5037.97+2.90 [−1.72, +8.74]
The translator contributed

Every NewsQA context exceeded the 128-key budget. Adding translated K/V improved F1 by 2.68 points; the interval, grouped by document, excludes zero. The learned map did add useful information.

Most of the gap remained

Replay alone lost 11.24 points against native on NewsQA. Translation recovered about 24% of that loss and still finished 8.56 points behind. Improving the translator alone would not explain away the larger replay deficit.

Keeping every key was exact

On all 189 contexts that fit within 128 keys, replay without injection produced exactly the native tokens. Differences appeared when the method actually removed keys.

What this leaves open: selecting the wrong keys and changing how later layers attend to them are still entangled. Even exact K/V from full prefill may not fit the state created by reduced replay, which is why O is not a ceiling. The layer-23 injection has no downstream prefix-cache layer; only the first three sites can change the returned cache. Other T−R intervals cross zero, SQuAD has only four context clusters, and several classification baselines are weak or produce invalid answers. We stopped this recipe here. This experiment measured answer quality, not serving speed or memory savings.

RaReCache: choosing the repair helps, but how much is enough?

Our independent reconstruction tested Qwen3-0.6B to Qwen3-14B on all 1,319 GSM8K math questions. Repairing more of the receiver's cache raised accuracy. Choosing what to repair by rank also beat a uniform choice with the same repair count. Both are useful results. The remaining question was whether either choice came close enough to the receiver's own cache.

Fixed repair budgets on the same GSM8K test set. Accuracy uses the frozen explicit-final scorer.
PathCorrect / 1,319AccuracyMeaning
Native Qwen3-14B1,25394.9962%The matched receiver reference.
Rank-selected 30% repair1,14086.4291%The lower repair budget.
Rank-selected 50% repair1,20691.4329%More repair helps, but the quality requirement still fails.
Uniform 50% repair1,18189.5375%Same repair count, simpler selection.

At 50% repair, rank selection gained 1.8954 percentage points over uniform selection, with a paired 95% interval of +0.2274 to +3.5633. It still trailed native by 3.5633 points. The predeclared rule required the one-sided 95% upper bound on that loss to stay within 3 points; the measured bound was 4.6247. Both 50% variants missed the quality rule.

A partial result, not a near-native reproduction. The paper reports 95.3% at 50% repair and 95.1% for native. Our native anchor is close; our repaired result is lower. Differences include prompting, calibration, source-layer selection and sharing, and scoring. We have not isolated which difference explains the gap. The fixed configuration is closed, and these diagnostic runtimes do not establish a serving speedup.

Can native recomputation rescue a translated cache?

DroidSpeak makes this an attractive idea, but the repair location and model relationship matter. Its positive result used two adapters on the same base model. Our cross-width Qwen3 test rebuilt the receiver's first layers and used frozen KV-Lingo maps above them. That is a different experiment.

The tested prefix repair fell short

Rebuilding the first 2, 4, 8, or 12 layers did not pass the quality rules in both directions on the 32-conversation development screen. The 4B→8B deficits were 13.07–16.35 F1 points against a 3-point limit. In reverse, the best gain over translation alone was just 0.52 points.

No boundary was selected, so timing, repeated handoffs, and fresh confirmation did not run. More depth search with these frozen maps is not the next step.

A middle-layer bridge is still a hypothesis

Rebuilding a middle interval would need a compatible hidden state at its entrance. Different-width models need an explicit bridge for that state, as well as layer alignment. The prefix test did not evaluate such a bridge, and the DroidSpeak adapter result does not prove it will work.

A new test should show what this added machinery buys over the receiver's exact cache and simpler repair controls before scaling up.

The prefix-repair figures summarize an independent experiment. Its exact run source and raw records are not published; this section is not an exact-replay package.

Where LMCache fits

Once a cache is compatible, it still needs to reach the process that will use it. LMCache handles that storage and movement. Our same-model control retrieved 7,168 cached tokens and produced an identical completion. A handoff to a different model adds another job: making the incoming state usable by that receiver.

Transport

LMCache

Names, stores, moves, and retrieves compatible KV data across processes or nodes.

Representation repair

KV-Lingo

Maps cached keys and values from one model representation to another.

Representation repair

DroidSpeak

Reuses compatible sender layers and rebuilds one interval with receiver compute.

Representation + replay

Modified MoT

Stores source state and a selection ledger, then runs receiver replay with translated injections.

The practical combination: use a cache service to carry the tensors, and a qualified receiver path to translate or rebuild them. The serving engine must check the model pair, token positions, selected keys, and replay recipe before reuse. Keeping an exact cache for each receiver remains the quality reference when comparing storage and latency costs.

Can we learn the next lesson without a large GPU run?

That is the next research question. Prefix Integrity Analysis already checks cache identity and block-position contracts on a CPU without loading a model. Those manifest checks do not establish that real cache tensors reload correctly or preserve an answer. We want to add a second cheap screen: small, trained models whose saved checkpoints let us test whether sharing preserves an answer.

Catch broken cache contracts. Reuse the existing CPU checks for identity, geometry, and query-dependent state. A cache that silently changes meaning between requests has failed before answer quality enters the picture.
Test with capable small models. Train compact models on controlled retrieval and conversation tasks, freeze their checkpoints, and compare native caches, translation, replay, and repair. Include both handoff directions and repeated switches.
Check whether the screen predicts anything. Compare its outcomes with known larger-model successes and failures, then test new methods on untouched cases. Only a calibrated screen should influence which expensive reproductions we run next.

Proposed, not yet validated. Small models can represent attention layouts and cache mechanics without representing all the behavior of a large language model. A toy success is not a paper reproduction, and a toy failure is not a refutation. The value of this screen depends on demonstrating which findings transfer across scale.

Try the modified-MoT implementation

The code is available if you want to inspect the mechanism or run your own comparison. This is the modified path discussed above, not a claim to reproduce the original paper. Use Python 3.10 or 3.11 with the pinned MoT requirements, and keep model snapshots and generated artifacts outside the source trees. The MoT fork retains the upstream PolyForm Noncommercial License.

Pin both repositories. The modified source commit is public on a dedicated branch. The knlp wrapper checks the exact seven-commit chain and key file hashes.
Run code and mechanics controls. Unit tests need no model. Real-model controls require the pinned GPT-2 and GPT-2-medium snapshots recorded in recipe.py.
Train and evaluate explicitly. Freeze the data manifest, run the diagnostic controls, train one directed checkpoint, then run the paired evaluation and four-arm attribution.
git clone https://github.com/mcgrof/knlp.git
git clone https://github.com/mcgrof/MoT.git modified-MoT
git -C modified-MoT checkout 9fb5e6f2a63981a53da1769c72e000a840ac976b

python3.11 -m venv mot-env
mot-env/bin/python -m pip install -r modified-MoT/requirements.txt pytest zstandard

cd knlp
../mot-env/bin/python -m pytest -q tests/kv_translate/test_mot_author.py
make defconfig-mot-author-readiness
make MOT_SOURCE=../modified-MoT MOT_AUTHOR_PYTHON=../mot-env/bin/python

The real-model mechanics gate checks native continuation, full-key replay, selected-key replay, token identity, and source provenance before training:

make defconfig-mot-author-mechanics
make mot-author-mechanics-preflight \
  MOT_VARIANT=modified-mot \
  MOT_SOURCE=../modified-MoT \
  MOT_AUTHOR_PYTHON=../mot-env/bin/python \
  MOT_SOURCE_MODEL=/path/to/gpt2-607a30d783dfa663caf39e06633721c8d4cfcd7e \
  MOT_TARGET_MODEL=/path/to/gpt2-medium-6dcaa7a952f72f9298047fd5137cd6e4f05f41da \
  MOT_OUTPUT=/path/to/new-run/mechanics

After you create a frozen manifest and checkpoint with the documented diagnostic targets, run the same four-arm evaluator:

make defconfig-mot-modified-attribution
make mot-author-attribution-eval \
  MOT_SOURCE=../modified-MoT \
  MOT_CHECKPOINT=/path/to/checkpoint \
  MOT_CHECKPOINT_SHA256=<translator-file-sha256> \
  MOT_DATA_MANIFEST=/path/to/diagnostic-data.json \
  MOT_ANCHOR_RECORDS=/path/to/paired-records.jsonl.zst \
  MOT_OUTPUT=/path/to/new-run/attribution
Available in the repositories

The exact source chain, configuration, data-free controls, model-revision checks, training path, evaluator, offline replay, and aggregate calculation are public.

Needed for an exact replay

The tested translator checkpoint, frozen 500-item manifest, and token-level anchor records are not in the public repository. You can reproduce the procedure with newly generated inputs, but not byte-replay this exact result without those artifacts.

Start with the full runbook: research/kv_translate/mot_author/README.md. It includes data freezing, training smoke, full training, paired evaluation, and offline replay commands.

Follow the evidence

This page contains static aggregate values. It does not load private data or machine-readable experiment artifacts.