Trained cartridges · asymmetric KV · fused serving

Compress the shared cartridge.
Keep the keys.

MMLU (Massive Multitask Language Understanding) samples many academic subjects to test broad knowledge and reasoning; GSM8K (Grade School Math 8K) uses math word problems to test multi-step reasoning. Together they are useful, complementary checks of general model quality, although these benchmark runs do not by themselves certify cartridge quality.

Cartridges expose a quantization penalty that those ordinary checks miss. In ordinary Qwen3-8B inference—no cartridge, unchanged model weights, and only the runtime KV cache changed—symmetric FP8 raises the measured MMLU score from 77.360% to 77.458%: a tiny +0.099 percentage-point improvement in this run. It lowers GSM8K from 93.0% to 91.5% (−1.5 points). Apply the same symmetric FP8 fake quantization after training to an existing cartridge, however, and its evaluation loss is 0.873–1.303% higher than the same cartridge in BF16. Quantizing the keys causes almost all of this penalty. K16/V8 keeps keys in BF16 and quantizes only values, holding the loss increase to 0.029–0.038% while reducing the raw KV payload by 25%.

What cartridge evaluation loss measures. During self-study, Qwen3-8B reads the full source document and answers synthetic questions about it. At each answer position, the run saves the probabilities the model assigns to possible next tokens. The full-document run is the teacher; those saved next-token probabilities are the teacher targets. Evaluation gives the same frozen model the cartridge instead of the full document and measures how closely its probabilities reproduce the saved ones. Lower loss means a closer match. This is a paired, post-training, in-sample score—not MMLU or GSM8K accuracy and not the trainer's final logged loss. The +0.099-point MMLU movement and the +1.303% cartridge-loss increase use different units; they are not endpoints on one percentage scale.
1 · Ordinary Qwen3-8B inferenceSymmetric FP8 K8/V8 runtime KV versus BF16 K16/V16. No cartridge; model weights stay unchanged. Bars show benchmark-score change in percentage points.
MMLU+0.099
GSM8K−1.500
−1.5BF16+1.5 pp

MMLU moves slightly upward. +0.099 points is roughly one additional correct answer per 1,000 questions if interpreted literally; one run does not establish that FP8 improves the model. GSM8K records three fewer correct answers out of 200.

2 · Existing cartridges expose the penaltyPost-training loss increase from fake-quantizing each saved BF16 cartridge. Lower is better.
V8 only0.029–0.038%
K8 only0.837–1.271%
K8 + V80.873–1.303%
0%+0.675%+1.35%
three-cartridge rangemean

The ordinary benchmarks above hide this cartridge-specific cost. Symmetric K8/V8 raises loss on every cartridge and reaches +1.303%; K8 alone lands in nearly the same range, so keys cause almost all the damage.

3 · Cartridge object sizeRaw payload for one 632-position Qwen3-8B cartridge. Scale and serialization metadata excluded.
BF1688.875 MiB
K16/V866.656 MiB
K8/V1666.656 MiB
K8/V844.438 MiB
044.488.9 MiB

K16/V8 saves 22.219 MiB (25%) per cartridge. Green marks the selected layout that retains BF16 keys. Red marks layouts that quantize keys and incur the loss penalty shown in panel 2.

This is post-training quantization. Start with an existing BF16 cartridge, leave its keys unchanged, and convert only its values to FP8 E4M3. The fake-quant experiment measures the numerical effect of that conversion on already-trained cartridges. The fused serving path stores the values as actual eight-bit data and converts them inside attention. Cartridge training is unchanged.

The deployment path is real, not fake quantization: a shared K16/V8 cartridge stays outside every request's live cache, and one FlashInfer FA3 kernel attends over the cartridge and live suffix with one online-softmax state. After correcting the JIT source binding and an odd-boundary warp bug, the native path passes every tested cartridge length and edges out ordinary BF16 cartridge injection in the controlled batch-1 and batch-16 runs.

Deployment resultUse a shared K16/V8 cartridge over an otherwise BF16 cache. On Qwen3-8B with the 615-position patient-01 cartridge, native K16/V8 raises median output throughput by 0.42% at batch 1 and 0.52% at batch 16, while median TTFT falls by 2.92% and 3.96%. Treat these as modest measured wins, not a broad GPU or workload guarantee.
ONE ATTENTION PASSPREFILL + DECODE20 / 20 SHAPE CASES PASSH100 / SM90

The fused path wins against a regular BF16 cartridge

Qwen3-8B on one NVIDIA H100 80 GB, CUDA 12.9, the exact 615-position patient-01 cartridge, vLLM eager mode, and FlashInfer FA3. Generate 32 tokens per request, run one warmup, then report the median of three timed repetitions. Both arms use the same model, prompts, live BF16 KV cache, and exact cartridge length; only the cartridge representation and attention path change.

Batch 1
+0.42%

BF16 injection reaches 88.27 output tok/s; native K16/V8 reaches 88.64. Median TTFT improves from 25.01 to 24.28 ms (−2.92%).

Batch 16
+0.52%

BF16 injection reaches 1,162.55 output tok/s; native K16/V8 reaches 1,168.57. Median TTFT improves from 77.94 to 74.85 ms (−3.96%).

batchordinary BF16 cartridgeshared K16/V8 cartridgetok/s deltaordinary TTFT p50shared TTFT p50TTFT delta
188.27 tok/s88.64 tok/s+0.42%25.01 ms24.28 ms−2.92%
161,162.55 tok/s1,168.57 tok/s+0.52%77.94 ms74.85 ms−3.96%
Read the result against the correct baseline. This table compares a shared K16/V8 cartridge plus BF16 live KV with ordinary BF16 cartridge injection plus BF16 live KV. The fused path wins on both reported throughput and TTFT, although the margin is small enough that additional GPUs and workloads remain necessary before making it a serving default.
Long decode check
82.23 tok/s

One batch-1 request sustains 82.23 output tok/s across 1,024 generated tokens. This confirms stable long decode; no matching BF16 long-run control was collected, so it is not a speedup claim.

Peak allocated HBM
−27.1 MiB

The measured engine peak is 71,129,956,864 bytes for native K16/V8 versus 71,158,363,648 bytes for BF16 injection after warmup. Allocator-wide peaks are nearly identical; the object-level 25% cartridge saving remains the precise capacity result.

When the live cache already uses K16/V8

The same fused kernel accepts FP8 values in each request's live cache as well as in the shared cartridge. Prefill and decode pass for FP8 live values at all five deployed cartridge lengths. That establishes numerical execution coverage, not a throughput result. The corrected source-bound kernel has not yet received a controlled performance rerun against the ordinary global-K16/V8 engine, so this page makes no speed claim for the globally asymmetric configuration.

Keys carry the cartridge quantization loss

Qwen3-8B has 36 layers, eight KV heads, and head dimension 128. The patient-02 cartridge has 632 positions: one frozen attention sink—a fixed leading KV position that every later token can attend to—and 631 trainable positions. Report raw KV payloads without scale or serialization metadata.

What loss means here. During self-study, Qwen3-8B sees the full source document and records the probabilities it assigns to possible next tokens while answering synthetic questions. That full-document run is the teacher, and its saved probability distributions are the teacher targets. After training, run the frozen model with the saved cartridge instead of the full document over a fixed slice of the same conversations. The distillation loss increases when the cartridge-run probabilities move away from the saved full-document probabilities, so smaller is better. This is a separate post-training evaluation, not the final loss value printed during training. These cartridges trained on the full self-study file, so the slice is in-sample, not held out. The BF16-versus-quantized comparison remains paired: each row evaluates the same cartridge on the same conversations.
layoutbytes / tokenfull 12,628-token KV632-position cartridgecapacityin-sample eval loss increase
BF16 K16/V16144 KiB1.734 GiB88.875 MiB1.00×baseline
Asymmetric K16/V8108 KiB1.301 GiB66.656 MiB1.333×+0.031%, +0.029%, +0.038%
Reverse asym. K8/V16108 KiB1.301 GiB66.656 MiB1.333×+0.837%, +1.271%, +1.140%
Symmetric K8/V872 KiB0.867 GiB44.438 MiB2.00×+0.873%, +1.303%, +1.177%
V8 only
+0.033%

Mean relative objective increase across the three separately trained cartridge checkpoints.

K8 only
+1.083%

Quantizing keys causes essentially all of the measured cartridge loss.

K8 + V8
+1.118%

Symmetric FP8 gains 2× payload capacity but misses the cartridge quality knee.

What the three loss values mean

trained cartridgeBF16 eval lossK16/V8 increaseK8/V8 eval lossK8/V8 increase
CAS cartridge run 10.016542+0.031%0.016687+0.873%
Half-learning-rate ablation0.016799+0.029%0.017018+1.303%
CAS cartridge run 20.016932+0.038%0.017131+1.177%

These are three separately trained cartridge artifacts, not three samples from one inference run. The half-learning-rate arm is an ablation; run 2 is a second CAS-regime cartridge. All loss values in this table are post-training, in-sample evaluations.

Why ordinary inference did not warn us

measurementBF16 K16/V16asymmetric K16/V8symmetric K8/V8
Regular inference MMLU77.360%77.393% (+0.033 pp)77.458% (+0.099 pp)
Regular inference GSM8K (n=200)93.0%92.5% (−0.5 pp)91.5% (−1.5 pp)
In-sample cartridge eval lossbaseline+0.029% to +0.038%+0.873% to +1.303%

The ordinary full-model benchmarks do not reveal the cartridge-specific penalty. In this run, symmetric FP8 moves the measured MMLU score upward by 0.099 points—too little evidence to conclude that quantization improves the model—while GSM8K loses three correct answers out of 200. The paired cartridge evaluation consistently detects the cost of quantizing keys. Treat a cartridge as its own quantization qualification workload; ordinary generation scores alone do not establish cartridge safety.

Five-patient fake-quant generation gate

Score 20 deterministic LongHealth multiple-choice questions for each of five already-trained cartridges. These arms use BF16 execution; the fake-quant arms first round the stored cartridge to the named FP8 layout and then reconstruct the rounded values in BF16. This tests numerical sensitivity, not physical FP8 storage or the fused kernel.

armpatient 01patient 02patient 03patient 05patient 06totalvs BF16
BF16 K16/V1610 / 2014 / 2013 / 2012 / 2014 / 2063 / 100baseline
Fake K16/V8, per-tensor scale9 / 2015 / 2013 / 2012 / 2011 / 2060 / 100−3
Fake K16/V8, unit scale11 / 2013 / 2011 / 2012 / 2011 / 2058 / 100−5
Fake K8/V8, per-tensor scale10 / 2013 / 2012 / 2011 / 2014 / 2060 / 100−3
The 100-question accuracy gate is coarser than the paired loss measurement. Per-tensor K16/V8 and K8/V8 both land at 60 / 100 here, while the continuous teacher-target loss cleanly separates their numerical damage: about +0.03% for K16/V8 versus +0.9–1.3% for K8/V8. Use the loss study to identify the key-side effect and the generation gate as a small downstream behavior check.

The corrected native path matches the BF16 decisions

Run the already-trained 615-position patient-01 cartridge through real vLLM generation. The native arm stores its values as FP8 E4M3 and dequantizes them inside the fused attention kernel; the fake arms reconstruct FP8-rounded values in BF16 containers before serving. Score the same 20 LongHealth multiple-choice questions with deterministic greedy decoding.

patient-01 armstorage / executionLongHealthexact sequence vs fixed nativesame correct / incorrect decision
BF16 cartridgeBF16 K / BF16 V injection10 / 2012 / 2020 / 20
Fake K16/V8, per-tensor scaleFP8-rounded V reconstructed in BF169 / 2011 / 2019 / 20
Fake K16/V8, unit scaleFP8-rounded V reconstructed in BF1611 / 2015 / 2019 / 20
Fake K8/V8, per-tensor scaleFP8-rounded K and V reconstructed in BF1610 / 203 / 2016 / 20
Native fused K16/V8BF16 K / physical FP8 V10 / 20referencereference
The native run is deterministic and restores BF16-level decisions on this slice. A warm rerun reproduces all 20 native token sequences exactly. Native and BF16 produce different wording on eight questions, but they mark the same ten questions correct and the same ten incorrect. This is one patient's 20-question slice, not the full five-patient quality gate.
Separate compilation time from generation time. The first corrected native run takes 137.56 seconds because it compiles the FlashInfer JIT kernel inside the timed region. The warm rerun takes 23.13 seconds; the BF16 run takes 23.50 seconds. Their generated lengths differ—2,851 native tokens versus 3,787 BF16 tokens—so those wall times validate warm execution but do not form a throughput comparison.

What counts as an acceptable accuracy tradeoff?

There is no universal point budget. Set the limit per workload, using enough samples to resolve the difference, and require the deployment benefit to justify it. Two public industry examples put the numbers in perspective.

Industry benchmark yardstick

MLPerf uses 99% and 99.9% accuracy gates

MLPerf Inference sets workload-specific floors. Its Mixtral question-answering, math, and code workload—which includes GSM8K—requires 99% of the FP16 reference. Several other workloads provide both 99% and stricter 99.9% tracks.

Borrow that rule only as an illustrative bar for these Qwen3-8B results; this is not an MLPerf submission.

MMLU K8/V8100.13%
GSM K16/V899.46%
GSM K8/V898.39%
99% reference99.9% reference

Scale spans 98% to 100.25% of each BF16 score. On this borrowed yardstick, K16/V8 clears 99% on both benchmarks; symmetric K8/V8 misses it on GSM8K.

Released mobile-model tradeoff

Meta released a mobile model with larger point drops

Meta's released Llama 3.2 3B SpinQuant model moves MMLU from 63.4 to 62.0 (−1.4 points) and GSM8K from 77.7 to 75.7 (−2.0 points).

OnePlus 12 resultSpinQuant vs BF16
Decode rate2.6×
Model file60.3% smaller
Resident memory49.8% lower

Meta reports that tradeoff for constrained on-device inference. It quantizes weights and activations, so it is a concrete industry precedent, not a directly comparable KV-only result.

Decision for this cartridge path. The symmetric-FP8 GSM8K result is only three answers out of 200 and needs a larger run before treating −1.5 points as stable. Even if that ordinary-model drop proved acceptable, symmetric FP8 still raises cartridge evaluation loss by 0.9–1.3%. K16/V8 is the stronger measured trade: 25% fewer cartridge bytes, about 0.03% higher cartridge loss, and a 0.5-point GSM8K change that clears the illustrative 99% bar.
The full-document entries report size, not quality. The 1.734, 1.301, and 0.867 GiB entries are byte accounting for a 12,628-token KV cache. The quality experiment quantizes trained cartridges, not the full document KV. The earlier 631-token initialization control used only the document's first 631 trainable positions and therefore does not measure full-document cache quality.

What fake quantization does with the other eight bits

The loss study rounded each BF16 value to the nearest FP8-E4M3 representable value, then stored that reconstructed number in a BF16 tensor so the existing model code could consume it. The BF16 container still occupies 16 bits, but the lower precision bits contain no independent information. They are fixed by the exact FP8 value represented in BF16; the conversion does not invent eight bits of new attention data.

The arithmetic uses the reconstructed numerical value. It does not ignore half of a BF16 register, but only the FP8-level information survives the round trip. Therefore fake quantization measures numerical damage and saves no memory. The serving kernel below stores a real eight-bit V payload and converts it while loading into the attention pipeline.

One kernel spans cartridge and live KV

Avoid the original two-attention-call design and its exact LSE merge. Present the shared cartridge and each request's paged live suffix as one logical sequence, then feed both regions into FlashInfer's existing register-resident online softmax.

Shared prefixRead dense BF16 keys and packed FP8-E4M3 values from one resident cartridge.
BoundarySelect cartridge or live storage only in the tile that crosses the cartridge's exact length.
Live suffixRead ordinary paged BF16 KV or paged K16/V8 KV for each request.
One outputKeep max, normalization sum, and output accumulator in registers; write once.
DENSE CARTRIDGE TILES

Bypass page-table division and index loads. Issue FP8 value reads before conversion and shared-memory stores to expose memory-level parallelism.

ONE ONLINE SOFTMAX

One attention launch avoids a second output, an LSE tensor, a temporary merge buffer, and a separate merge kernel. This specialization also disables internal split-KV partial outputs.

DECODE GEOMETRY

Use a 64×128 tile with one consumer warpgroup. Give producer and consumer 216 registers each; the final kernel uses 32 bytes of stack.

PREFILL GEOMETRY

Retain the 128×96 prefill tile. The custom dtype pair and mixed-source planning cover both prefill and decode.

Two fixes made the native result real

BIND JIT TO THE TESTED SOURCE TREE

The editable FlashInfer installation on gpu1 still pointed its JIT include and source symlinks at an obsolete July checkout. The benchmark driver came from the fused branch, but runtime compilation consumed headers that predated the cartridge-aware mainloop. Binding the JIT paths to the same 68153db5-based tree makes the compiled kernel match the code under test.

CONVERGE THE ODD-BOUNDARY SHUFFLE

A 615-token boundary splits a warp between cartridge values and live values. The cartridge-value offset used __shfl_sync(FULL_MASK) only inside the cartridge branch, so lanes named by the mask did not all execute the shuffle and the warp could deadlock. Compute both source offsets before the branch; every lane now executes both full-mask shuffles.

Exact lengths now work. The kernel passes prefill and decode with both BF16 and FP8 live values at the aligned 632-token/page-8 shape and the odd 615-token/page-1 shape: 8 / 8 cases. Repeating that four-way matrix for all five deployed lengths—615, 632, 644, 506, and 566—passes 20 / 20 production-shape cases.

Memory behavior

Packed cartridge
66.656 MiB

K16/V8 payload for one 632-position Qwen3-8B cartridge, down from 88.875 MiB BF16.

Avoided at batch 64
5.49 GiB

One shared cartridge replaces 64 copies of an 88.875 MiB BF16 prefix in the requests' ordinary caches.

Global K16/V8 pool
+33.33%

Reported live KV-token capacity rises from 365,200 to 486,928 when the whole cache uses K16/V8.

The implementation is split between FlashInfer and vLLM

FlashInfer owns dtype-specialized attention and the fused mixed-source kernel. vLLM owns cartridge residency, scheduler allocation, exact-length planning, backend dispatch, and the end-to-end benchmark. The odd-boundary repair changes only FlashInfer; the existing vLLM integration runs unchanged once its JIT points at the repaired FlashInfer tree.

FlashInfer · 3-commit cartridge series

Review the repaired FlashInfer branch

8b3608e6 lets custom decode and prefill modules compile distinct key and value dtypes.

68153db5 fuses the dense shared cartridge with paged live KV inside one Hopper FA3 attention pass.

fd1a21de keeps full-mask source-offset shuffles converged across odd cartridge boundaries and adds the 615-token regression shape.

vLLM · 5-commit serving series

Review the exact-length vLLM branch

1dfe7de2e0 preserves split K and V planes in GPU residency.

c04e7137d1 releases cache blocks owned by the external cartridge.

0ec079cdf1 plans and serves the cartridge and live suffix in one pass.

a7cb3769f5 adds the fresh-process four-layout benchmark and summary.

627eea6875 preserves exact cartridge lengths with one-token FlashInfer pages, including the 615-, 644-, 506-, and 566-position artifacts that do not divide by eight.

No KNLP runtime patch is required. KNLP holds the research harness and public analysis. The executing code lives in the FlashInfer and vLLM branches above.

Validation gates

gatecoverageresult
FlashInfer boundary matrixprefill/decode × BF16/FP8 live V; aligned token-632/page-8 and odd token-615/page-1 boundaries8 / 8 pass
Deployed-length matrixthe same four cases at 615, 632, 644, 506, and 566 cartridge positions20 / 20 pass
vLLM connector, core, fused attentionresidency, allocation, routing, planning, backend dispatch81 / 81 pass
Static checksRuff, formatter, git diff whitespacepass
Real vLLM generationpatient-01, 20 deterministic LongHealth questions; compare native K16/V8 with BF16 injection10 / 20 each; decisions match 20 / 20
Native reproducibilityrepeat the corrected native patient-01 run from a warm cache20 / 20 token sequences match

The three-checkpoint objective study remains the strongest measured quantization signal. The patient-01 generation result validates native execution against BF16 on one 20-question slice; a corrected five-patient native rerun remains necessary for a full generation-quality estimate.

Keep the result inside its tested envelope

HOPPER ONLY

The fused specialization targets SM90 / FlashInfer FA3. An AMD W7900 can reproduce the fake-quant numerical study, but it cannot compile or execute this CUDA/Hopper kernel. Ampere and ROCm performance remains unmeasured.

EAGER EXECUTION

The current vLLM path excludes CUDA graphs. Measure graph-compatible integration separately.

ONE HOMOGENEOUS CARTRIDGE

Routing supports a singleton shared cartridge per request group. Multi-cartridge composition is not implemented in this specialization.

EXACT PREFIX LENGTHS

One-token FlashInfer pages preserve non-divisible lengths. The tested 615-, 632-, 644-, 506-, and 566-position cartridges all pass prefill and decode with BF16 and FP8 live values.

NO DCP OR SPECULATION

Data-context parallelism and speculative decoding are outside the current contract.

FP8 E4M3 VALUES

The packed cartridge path supports BF16 keys and FP8-E4M3 values. Other value formats need their own numerical and performance gates.

Next qualification target. Repeat the controlled batch sweep against ordinary global K16/V8 with the repaired source-bound kernel, then run the corrected native arm across all five LongHealth patients. Those two measurements decide whether shared asymmetric cartridges should become the default when the model's live cache is already asymmetric.