Cartridges at Scale

1. What a cartridge is, and what CAS adds

A cartridge is a trained KV cache. Train it once per document offline, inject it into the model's KV cache at serving time, and skip prefill. The base technique trains that prefix by self-study context distillation: a teacher answers synthetic questions about the document, and the cartridge is trained to match the teacher's next-token distribution. One document, one cartridge, one prefix at serving time.

Cartridges at Scale keeps that per-document cartridge and adds four things:

• Scale to a collection. One cartridge per document, many resident on the GPU at once, any subset composed at request time.
• A budget manager. During training, keep a fixed number of cartridges on the GPU and rotate the active set every few steps so a large collection trains within memory.
• Mixed-visibility joint training. Independently trained cartridges interfere when placed in one cache. CAS co-trains them with a per-sample visibility rule so they learn to coexist.
• Collapse and rescue. The CAS paper reports that isolated cartridges collapse toward the no-context floor when co-loaded and that the joint rule prevents it.

2. What is validated

The validated result is the isolated-cartridge reproduction. Accuracy uses the CAS Table-15 LongHealth protocol: option-text prompting, thinking enabled, temperature 0.6, a 2048-token generation budget, fuzzy option matching, all 20 questions for each patient, and three generation runs per trained cartridge.

The no-context and full-document anchors reproduce the paper within about two percentage points. The exact full-document KV control proves that the alternate cache path itself does not explain the learned-cartridge gap. The five-patient number is a mean over five separately trained H100 cartridges. Each patient score averages three sampled generations of the same trained cartridge; those three generations do not measure training-to-training variation.

This reproduction measures isolated-cartridge training only. It does not include cartridge co-loading or mixed-visibility joint-training results.

3. The faithful isolated recipe

Holding the patient and data fixed, replacing the small-batch optimizer regime with the paper's global-batch-128 schedule raised patient 02 from 0.55 to 0.65 ± 0.05. Batch size, learning rate, and schedule changed together, so this comparison establishes the value of the complete regime rather than attributing the gain to one component.

The schedule horizon is not the number of optimizer updates. For patient 02, 80 epochs end at roughly 1,000 updates while the learning rate is still about 0.083. The harness now stops explicitly at a requested step; the upstream trainer's max_optimizer_steps hook saves a checkpoint but does not otherwise stop training.

4. Five-patient isolated result

Five separate cartridges were trained with the faithful H100 recipe. The 2,048-token column follows the paper's completion limit. The 8,192-token column re-evaluates the same cartridge and questions with enough room for every answer to finish. Both columns use Qwen3's shipped top-k 20 and top-p 0.95 sampler.

The five-patient mean remains below the paper's isolated aggregate of 0.736. The comparison also shows that completion length is not one general explanation for the gap: patient 03 improves sharply, while the other four patients move by at most one answer across their 60 sampled generations.

5. Completion-cap diagnostic

A completion counts as capped only when re-tokenizing its stored text reaches the configured token limit. The evaluator's older truncated field instead recorded whether the model closed its thinking block; treating that field as a cap counter incorrectly marked 17 patient-01 answers even though none reached 2,048 tokens.

The longer cap recovers eleven answers for patient 03 and one for patient 06, while patient 02 loses one through ordinary sampling variation. It raises the five-patient mean by 0.037, leaving most of the distance to 0.736 unexplained. The paper synthesizes roughly 40,000 conversations per cartridge versus about 4,400 here; these measurements do not isolate that data-scale difference.

6. Training-run variation

Eight new patient-02 trainings isolate reproducibility from generation noise: two fixed-seed controls and six additional seeds. Every cartridge was evaluated in the same session with three 20-question runs and the corrected sampler.

The two seed-42 runs produce byte-identical cartridge files and identical generations. Fixed-seed training is deterministic on this stack. Across the seven distinct seeds, however, five cartridges score 0.750–0.800 and two score 0.400, close to the paper's 0.375 no-context result. The mean is 0.667 with a sample standard deviation of about 0.2. The two low-scoring runs also have clearly higher final training loss, identifying a failed training outcome rather than evaluation noise alone.

Common-session checks of existing cartridges

All four existing cartridges fall inside the observed seed range. The A100 cartridge exceeds the reference by two correct answers out of 60 in this session, with overlapping generation spread. The earlier 0.833 score therefore does not establish a hardware effect. Likewise, the learning-rate and prior same-recipe differences are smaller than training-seed variation.

7. The cartridge path is lossless

The cartridge evaluator uses a custom FlexAttention Qwen3 forward, so the ordinary full-document anchor alone cannot validate it. The direct control prefills the complete roughly 12,000-token record, captures every exact K and V tensor, writes that full KV into cartridge format unchanged, and evaluates it through the cartridge path. It scores 0.86, matching ordinary full-document inference at 0.855.

8. Training stability

Training seed controls the failure mode

The seven-seed run finds two distinct outcomes. Five seeds finish near loss 0.30 and score 0.75–0.80; two finish above loss 0.43 and score 0.40. Small single-cartridge ablations cannot be interpreted against that spread. Future comparisons need repeated trainings and a training-loss acceptance check; more documents are required to set a general threshold.

The clipping probe produced an unclipped replicate

The 0.1 peak learning rate uses plain Adam without gradient clipping. A 210-step patient-02 probe logged every gradient norm but did not reproduce the loss excursion that motivated a clipping test. The resulting guard threshold never activated, so the completed run is an ordinary same-recipe replicate, not evidence for or against clipping. That replicate's 0.750 score instead exposed the larger training-run variation described above.

9. Code and reproduction

The public research/cartridges_cas harness contains the synthesis, isolated and joint trainers, Table-15 evaluator, cache-path parity control, prompt-grouped validation split, and checkpoint controls.

Three rules are load-bearing:

• Generated-target validity. The writer enforces real token IDs and uniqueness within each row's top-k support.
• Cache reconstruction. An isolated checkpoint consists of one frozen sink plus trainable KV. A joint checkpoint's frozen region contains distractors; an oracle read consists of the target's trainable region and its sink representation.
• Protocol calibration. Interpretable cartridge evaluations place no-context near 0.39 and full-document inference near 0.855. The reported values average at least three runs because a single 20-question run is noisy.

Compiled FlexAttention on CUDA sm≥80 gives about a 16× training speedup over the raw path. The target flattener also preserves the full available top-k when cumulative probability does not reach its threshold, instead of silently falling back to top-1.

Five-patient H100 reproduction

cas-paper is the profile for the result on this page. It trains patients 01, 02, 03, 05, and 06, evaluates each cartridge at the 2,048- and 8,192-token caps, and runs exact full-document KV through the cartridge path.

make defconfig-cas-paper
CUDA_VISIBLE_DEVICES=0 \
PAPER_DATA_DIR=/path/to/per_patient \
RECORDS_DIR=/path/to/records \
OUT_DIR=/path/to/cas-output \
CART_ROOT=/path/to/cartridges \
PYTHON=/path/to/python \
HF_HUB_OFFLINE=1 make

Eight-training seed spread

cas-training-spread-h100 launches the two seed-42 controls and seeds 43 through 48 across eight H100s, evaluates all eight artifacts in the same software session, checks repeated-seed byte identity, and writes the summary to OUT_DIR/training_spread/results.json.

make defconfig-cas-training-spread-h100
PAPER_DATA_DIR=/path/to/per_patient \
RECORDS_DIR=/path/to/records \
OUT_DIR=/path/to/cas-output \
CART_ROOT=/path/to/cartridges \
PYTHON=/path/to/python \
HF_HUB_OFFLINE=1 make

The spread profile regenerates the eight new training rows above. The reference, half-learning-rate, prior-run, and A100 comparison rows require their separately archived cartridges. Use cas-paper-regime-a100 for a new single-patient A100 control.

10. Serving

The serving side is already built. A cartridge injects as a KVConnectorBase_V1 plugin into vLLM with no core modifications, and on this record the time-to-first-token drops from 757 ms (re-prefilling the ~12K-token document every query) to 81 ms (loading the 611-token KV prefix), a 677 ms median saving per query at equal decode throughput. The connector, the multi-cartridge router, GPU residency and eviction, and an asymmetric-KV storage codec have all been implemented and tested against vLLM and LMCache.

CAS makes that plumbing broadly useful: a document collection can be compiled once, kept resident under a budget, and selected per request without repeatedly prefilling each retrieved document. Quantization of the stored cartridge is a separate serving R&D result and is intentionally documented separately from this training reproduction.