Learned Prefix Caches
are the shared object.
Cartridges, ReasonCACHE, skill caches, and document caches all attach learned layer-wise KV prefixes to a frozen model. The real difference is how training behavior is induced.
KNLP uses Learned Prefix Cache (LPC) for the common artifact and Prefix Induction (PI) for the bridge process that turns raw documents, skill files, and demonstrations into trainable traces.
Learned Prefix Cache
Layer-wise learned K/V prefix, frozen backbone, prompt starts after the virtual prefix.
Cartridges
Raw corpus → SS-PI → context distillation → document prefix cache.
ReasonCACHE
Curated reasoning traces → next-token or prefix training → reasoning prefix cache.
AR-PI
Raw skills or documents → active-reading traces → KL or NTP → induced prefix cache.
The mechanism is not the missing piece.
The cleaner decomposition is source material, prefix induction, training objective, and learned prefix cache. That separates the research question from the serving primitive.
Learned Prefix Cache
The common artifact: learned per-layer K/V state prepended to each request. Cartridges, ReasonCACHE, skill caches, and document caches differ in payload and training pipeline, not serving shape.
Prefix Induction
The bridge process that turns raw source material into behavior the prefix can learn. Cartridges use fixed self-study. ReasonCACHE starts from traces and mostly skips induction.
Active Reading Prefix Induction
Active Reading generates task-specific study views, questions, procedures, and reasoning traces before prefix-cache training. This is the current active-reading PI candidate.
Cartridges solve a document induction problem.
The paper targets raw corpora. Naive next-token training learns to continue or memorize the document, so Cartridges add self-study conversations and context distillation to mimic a teacher that still has the corpus in context.
ReasonCACHE assumes the traces already exist.
The paper trains learned KV prefixes from reasoning demonstrations. That validates the mechanism for reasoning, but leaves open the practical question: how do we create reasoning traces when the source is only a skill file, manual, dataset, or document?
Active Reading fills one trace-induction hole.
Active Reading Prefix Induction is one bridge under test: generate trainable behavior from uncurated sources, then train the same learned prefix cache object used by Cartridges and ReasonCACHE.
The key mistake to avoid
A skill file is documentation, not a target transcript. Raw SKILL.md next-token training is a negative control because it teaches continuation of instructions, while the target behavior is task solving.
The stronger experiment
Hold the cache mechanism, model, prefix size, teacher, and scoring fixed. Compare fixed self-study PI against AR-PI, and compare next-token training against teacher-KL distillation.
PI quality and initializer choice both matter.
The H100 skill-trace pilot showed that generated behavior traces matter. The H100 stabilization and GSM8K sanity runs add the next split: curated reasoning traces make ReasonCACHE work, while raw skills need Prefix Induction before cache training is useful. The H100 public-strata scale run is the first negative transfer warning: CSB-PI and BO-PI did not generalize beyond the focused XLSX surface, though one public task is harness-confounded. The A100 runs added two more axes: fixed-budget initializer quality and checkpointed steps-to-quality. First-k, paper SCI, and legacy strided-RoPE are different initializers. The corrected LongHealth redo also separates skill-cache behavior from valid long-document cartridge behavior.
Aggregate pilot pass rate
Same four SkillsBench tasks. The R-PI arm is the best current synthetic trace arm.
Prompt cost versus quality
The current vLLM contract still uses placeholder tokens, so this is an implementation Pareto, not the final virtual-prefix cost model.
SkillsBench PI pilot table
The table keeps the induction target visible. The biggest signals are Excel table into PPT and recover missing XLSX data. Powerlifting and weighted GDP stayed flat, so the next run needs broader task strata and better error isolation.
| Task | Raw skill NTP | Fixed self-study KL | AR-PI skill KL | R-PI KL | Interpretation |
|---|---|---|---|---|---|
| Excel table into PPT | 0 / 8 | 0 / 8 | 7 / 8 | 7 / 8 | Active Reading converted static procedure into usable behavior. |
| Powerlifting coefficient calculation | 0 / 11 | 0 / 11 | 0 / 11 | 0 / 11 | Likely blocked by formula, verifier, or execution details. |
| Recover missing XLSX data | 1 / 19 | 12 / 19 | 1 / 19 | 14 / 19 | R-PI beats fixed self-study on this task. |
| Weighted GDP calculation | 10 / 27 | 10 / 27 | 10 / 27 | 10 / 27 | The shallow arms do not change the decisive bottleneck. |
Current frontier update
The newer frontier keeps the mechanism and the induction problem separate. ReasonCACHE works when traces are already present. Skill caches need an induction algorithm that turns source material into trainable behavior before the prefix is optimized.
| Surface | Family | Result | Request tokens | Prefix tokens | Read |
|---|---|---|---|---|---|
| XLSX recovery | BO-PI + KL | 17 / 19 | 4,072 | 1,744 | Best score, not stable enough yet. |
| XLSX recovery | CSB-PI + KL | 14 / 19 | 3,641 | 1,744 | Focused signal; did not transfer in scale. |
| XLSX recovery | SS-PI + KL | 12 / 19 | 3,505 | 1,744 | Cartridges-style induction baseline. |
| GSM8K | ReasonCACHE NTP | 20 / 24 | 251.6 mean | 512 | Curated traces make the mechanism useful. |
| GSM8K | No cache | 17 / 24 | 301.7 mean | 0 | Mechanism baseline. |
| Public skill strata | CSB-PI / BO-PI + KL | 0 / 48 each | varies | varies | Focused XLSX gains did not generalize; one slice needs harness repair. |
A100 retest: proper AR labels and real SCI separation
The follow-up run separated paper task-agnostic Active Reading (AR-TA), paper task-specific Active Reading (AR-TS), the older KNLP Active Reading prompt, paper SCI, first-k, and the legacy strided-RoPE initializer. That matters because the best xlsx row came from AR-TA + strided-RoPE, while paper SCI depended strongly on the sampled chunks.
| Task | Best A100 row | No-skill baseline | Interpretation |
|---|---|---|---|
| Excel table into PPT | 0 / 8 | 0 / 8 | No tested shallow arm solved this task in the A100 matrix. |
| Recover missing XLSX data | AR-TA + strided-RoPE: 15 / 19 | 6 / 19 | Proper paper AR-TA became the strongest cache-induction arm. |
| syzkaller ppdev syzlang | best learned-prefix rows: 1 / 7 | 0 / 7 | Weak signal, but first-k stayed below SCI and strided-RoPE. |
| xlsx SCI seed sweep | SCI seed 0 | SCI seed 1 | SCI seed 2 | Best non-SCI comparison |
|---|---|---|---|---|
| SS-PI + KL | 0 / 1 invalid | 13 / 19 | 13 / 19 | strided-RoPE: 13 / 19 |
| AR-TA + KL | 1 / 19 | 13 / 19 | 3 / 19 | strided-RoPE: 15 / 19 |
| AR-TS + KL | 0 / 1 invalid | 13 / 19 | 13 / 19 | strided-RoPE: 1 / 19 |
| BO-PI + KL | 0 / 1 invalid | 13 / 19 | 11 / 19 | strided-RoPE: 0 / 1 invalid |
A100 convergence curve: SCI is not the fast default
This table uses verifier quality by optimizer step, not training wall-clock. It answers the convergence question directly for the xlsx skill-cache task.
| PI | Initializer | Step to 12 / 19 | Step to 13 / 19 | Step to 15 / 19 | Best |
|---|---|---|---|---|---|
| AR-TA | first-k | 8 | 8 | 8 | 17 / 19 |
| AR-TA | SCI seed 0 | — | — | — | 8 / 19 |
| AR-TA | SCI seed 1 | 16 | 16 | — | 13 / 19 |
| AR-TA | SCI seed 2 | — | — | — | 8 / 19 |
| AR-TA | strided-RoPE | 4 | 4 | 4 | 16 / 19 |
| SS-PI | first-k | 2 | — | — | 12 / 19 |
| SS-PI | SCI seed 1 | 16 | 16 | — | 14 / 19 |
| SS-PI | SCI seed 2 | 4 | 4 | — | 14 / 19 |
| SS-PI | strided-RoPE | 8 | 8 | — | 13 / 19 |
LongHealth document curve: random SCI is seed-sensitive
The corrected long-context redo uses LongHealth patient-record text, not SkillsBench skill files. The A100 multi-document run selected four records with 9,603 to 12,503 wrapped source tokens and a 2,048-token learned KV prefix. Training data, objective, model, prefix length, and evaluator are fixed; only the initializer changes.
| Patient | Source tokens | First-k best | Best SCI | Delta | Verdict |
|---|---|---|---|---|---|
| patient_05 | 9,603 | 11 / 20 | 11 / 20 | 0 | Tie |
| patient_06 | 10,807 | 11 / 20 | 11 / 20 | 0 | Tie |
| patient_12 | 12,503 | 17 / 20 | 13 / 20 | -4 | First-k win |
| patient_18 | 11,904 | 8 / 20 | 10 / 20 | +2 | SCI win |
Skill files under test
| Task | Individual SKILL.md files | Loaded skill chars | ReasonCACHE prompt chars |
|---|---|---|---|
| Excel table into PPT | pptx: 25,534; xlsx: 5,243 | 30,812 | 850 |
| Powerlifting coefficient calculation | powerlifting: 14,857; senior-data-scientist: 5,630; xlsx: 5,243 | 25,808 | 5,221 |
| Recover missing XLSX data | data-reconciliation: 2,482; xlsx: 5,243 | 7,775 | 2,356 |
| Weighted GDP calculation | xlsx: 5,243 | 5,260 | 1,216 |
Historical cartridge baseline
These older cartridge results are useful baselines, not direct ReasonCACHE proof. They establish variance and the importance of the induction pipeline on the same skill surfaces.
| Task | Full skill | Truncated skill | No skill | Cartridge 50% |
|---|---|---|---|---|
| Excel table into PPT | 0 / 8 | 7 / 8 | 7 / 8 | 7 / 8 |
| Powerlifting coefficient calculation | 9 / 11 | 5 / 11 | 5 / 11 | 5 / 11 |
| Recover missing XLSX data | 0 / 1 | 2 / 19 | 1 / 19 | 14 / 19 |
| Weighted GDP calculation | 1 / 27 | 1 / 27 | 1 / 27 | 1 / 27 |
The vLLM branch should be LPC support, not routing support.
The current support branch is based on the cartridge upstream work and deliberately avoids the routing branch. Routing can layer on later; the immediate primitive is externally supplied virtual-prefix KV blocks.
Serving contract
- prefix handle and resource kind;
- virtual token count and cache block alignment;
- layout, dtype, model identity, tokenizer identity, and RoPE identity;
- prompt positions and attention metadata account for the virtual prefix;
- LMCache can own storage, residency, and reuse policy.
Branch links
The public docs should point at the LPC branch because it carries Cartridges plus ReasonCACHE support without implying dependence on the routing work.
The strategic move is Prefix Induction.
This is work in progress. PI is now being treated like the KRI algorithm-vetting layer: hold the LPC mechanism fixed, compare task-oriented induction algorithms, and price each one by trace generation, teacher passes, prefix bytes, online prompt tokens, and expected requests sharing the learned prefix. The public-strata scale run showed the first promoted PI families are still task-local, with one public slice blocked by harness assumptions.
SS-PI
Self-Study Prefix Induction keeps the Cartridges-style baseline: generate broad synthetic conversations or QA around the source, then distill against the teacher behavior.
AR-TA / AR-TS
Active Reading Prefix Induction keeps the paper split between task-agnostic and task-specific strategy generation, then trains the learned prefix from induced traces.
R-PI
Reasoning Prefix Induction focuses the generated traces on intermediate reasoning, decision points, and task-solving rationales when curated traces do not already exist.
CSB-PI
Contract/self-check/bounded Prefix Induction repeated the xlsx gain without the BO-PI seed spike, but the H100 public-strata scale run did not reproduce that gain on broader skills.