Compress the shared cartridge.
Keep the keys.
MMLU (Massive Multitask Language Understanding) samples many academic subjects to test broad knowledge and reasoning; GSM8K (Grade School Math 8K) uses math word problems to test multi-step reasoning. Together they are useful, complementary checks of general model quality, although these benchmark runs do not by themselves certify cartridge quality.
Cartridges expose a quantization penalty that those ordinary checks miss. In ordinary Qwen3-8B inference—no cartridge, unchanged model weights, and only the runtime KV cache changed—symmetric FP8 raises the measured MMLU score from 77.360% to 77.458%: a tiny +0.099 percentage-point improvement in this run. It lowers GSM8K from 93.0% to 91.5% (−1.5 points). Apply the same symmetric FP8 fake quantization after training to an existing cartridge, however, and its evaluation loss is 0.873–1.303% higher than the same cartridge in BF16. Quantizing the keys causes almost all of this penalty. K16/V8 keeps keys in BF16 and quantizes only values, holding the loss increase to 0.029–0.038% while reducing the raw KV payload by 25%.
MMLU moves slightly upward. +0.099 points is roughly one additional correct answer per 1,000 questions if interpreted literally; one run does not establish that FP8 improves the model. GSM8K records three fewer correct answers out of 200.
The ordinary benchmarks above hide this cartridge-specific cost. Symmetric K8/V8 raises loss on every cartridge and reaches +1.303%; K8 alone lands in nearly the same range, so keys cause almost all the damage.
K16/V8 saves 22.219 MiB (25%) per cartridge. Green marks the selected layout that retains BF16 keys. Red marks layouts that quantize keys and incur the loss penalty shown in panel 2.
This is post-training quantization. Start with an existing BF16 cartridge, leave its keys unchanged, and convert only its values to FP8 E4M3. The fake-quant experiment measures the numerical effect of that conversion on already-trained cartridges. The fused serving path stores the values as actual eight-bit data and converts them inside attention. Cartridge training is unchanged.
The deployment path is real, not fake quantization: a shared K16/V8 cartridge stays outside every request's live cache, and one FlashInfer FA3 kernel attends over the cartridge and live suffix with one online-softmax state. After correcting the JIT source binding and an odd-boundary warp bug, the native path passes every tested cartridge length and edges out ordinary BF16 cartridge injection in the controlled batch-1 and batch-16 runs.
The fused path wins against a regular BF16 cartridge
Qwen3-8B on one NVIDIA H100 80 GB, CUDA 12.9, the exact 615-position patient-01 cartridge, vLLM eager mode, and FlashInfer FA3. Generate 32 tokens per request, run one warmup, then report the median of three timed repetitions. Both arms use the same model, prompts, live BF16 KV cache, and exact cartridge length; only the cartridge representation and attention path change.
BF16 injection reaches 88.27 output tok/s; native K16/V8 reaches 88.64. Median TTFT improves from 25.01 to 24.28 ms (−2.92%).
BF16 injection reaches 1,162.55 output tok/s; native K16/V8 reaches 1,168.57. Median TTFT improves from 77.94 to 74.85 ms (−3.96%).
| batch | ordinary BF16 cartridge | shared K16/V8 cartridge | tok/s delta | ordinary TTFT p50 | shared TTFT p50 | TTFT delta |
|---|---|---|---|---|---|---|
| 1 | 88.27 tok/s | 88.64 tok/s | +0.42% | 25.01 ms | 24.28 ms | −2.92% |
| 16 | 1,162.55 tok/s | 1,168.57 tok/s | +0.52% | 77.94 ms | 74.85 ms | −3.96% |
One batch-1 request sustains 82.23 output tok/s across 1,024 generated tokens. This confirms stable long decode; no matching BF16 long-run control was collected, so it is not a speedup claim.
The measured engine peak is 71,129,956,864 bytes for native K16/V8 versus 71,158,363,648 bytes for BF16 injection after warmup. Allocator-wide peaks are nearly identical; the object-level 25% cartridge saving remains the precise capacity result.
When the live cache already uses K16/V8
The same fused kernel accepts FP8 values in each request's live cache as well as in the shared cartridge. Prefill and decode pass for FP8 live values at all five deployed cartridge lengths. That establishes numerical execution coverage, not a throughput result. The corrected source-bound kernel has not yet received a controlled performance rerun against the ordinary global-K16/V8 engine, so this page makes no speed claim for the globally asymmetric configuration.
Keys carry the cartridge quantization loss
Qwen3-8B has 36 layers, eight KV heads, and head dimension 128. The patient-02 cartridge has 632 positions: one frozen attention sink—a fixed leading KV position that every later token can attend to—and 631 trainable positions. Report raw KV payloads without scale or serialization metadata.
| layout | bytes / token | full 12,628-token KV | 632-position cartridge | capacity | in-sample eval loss increase |
|---|---|---|---|---|---|
| BF16 K16/V16 | 144 KiB | 1.734 GiB | 88.875 MiB | 1.00× | baseline |
| Asymmetric K16/V8 | 108 KiB | 1.301 GiB | 66.656 MiB | 1.333× | +0.031%, +0.029%, +0.038% |
| Reverse asym. K8/V16 | 108 KiB | 1.301 GiB | 66.656 MiB | 1.333× | +0.837%, +1.271%, +1.140% |
| Symmetric K8/V8 | 72 KiB | 0.867 GiB | 44.438 MiB | 2.00× | +0.873%, +1.303%, +1.177% |
Mean relative objective increase across the three separately trained cartridge checkpoints.
Quantizing keys causes essentially all of the measured cartridge loss.
Symmetric FP8 gains 2× payload capacity but misses the cartridge quality knee.
What the three loss values mean
| trained cartridge | BF16 eval loss | K16/V8 increase | K8/V8 eval loss | K8/V8 increase |
|---|---|---|---|---|
| CAS cartridge run 1 | 0.016542 | +0.031% | 0.016687 | +0.873% |
| Half-learning-rate ablation | 0.016799 | +0.029% | 0.017018 | +1.303% |
| CAS cartridge run 2 | 0.016932 | +0.038% | 0.017131 | +1.177% |
These are three separately trained cartridge artifacts, not three samples from one inference run. The half-learning-rate arm is an ablation; run 2 is a second CAS-regime cartridge. All loss values in this table are post-training, in-sample evaluations.
Why ordinary inference did not warn us
| measurement | BF16 K16/V16 | asymmetric K16/V8 | symmetric K8/V8 |
|---|---|---|---|
| Regular inference MMLU | 77.360% | 77.393% (+0.033 pp) | 77.458% (+0.099 pp) |
| Regular inference GSM8K (n=200) | 93.0% | 92.5% (−0.5 pp) | 91.5% (−1.5 pp) |
| In-sample cartridge eval loss | baseline | +0.029% to +0.038% | +0.873% to +1.303% |
The ordinary full-model benchmarks do not reveal the cartridge-specific penalty. In this run, symmetric FP8 moves the measured MMLU score upward by 0.099 points—too little evidence to conclude that quantization improves the model—while GSM8K loses three correct answers out of 200. The paired cartridge evaluation consistently detects the cost of quantizing keys. Treat a cartridge as its own quantization qualification workload; ordinary generation scores alone do not establish cartridge safety.
Five-patient fake-quant generation gate
Score 20 deterministic LongHealth multiple-choice questions for each of five already-trained cartridges. These arms use BF16 execution; the fake-quant arms first round the stored cartridge to the named FP8 layout and then reconstruct the rounded values in BF16. This tests numerical sensitivity, not physical FP8 storage or the fused kernel.
| arm | patient 01 | patient 02 | patient 03 | patient 05 | patient 06 | total | vs BF16 |
|---|---|---|---|---|---|---|---|
| BF16 K16/V16 | 10 / 20 | 14 / 20 | 13 / 20 | 12 / 20 | 14 / 20 | 63 / 100 | baseline |
| Fake K16/V8, per-tensor scale | 9 / 20 | 15 / 20 | 13 / 20 | 12 / 20 | 11 / 20 | 60 / 100 | −3 |
| Fake K16/V8, unit scale | 11 / 20 | 13 / 20 | 11 / 20 | 12 / 20 | 11 / 20 | 58 / 100 | −5 |
| Fake K8/V8, per-tensor scale | 10 / 20 | 13 / 20 | 12 / 20 | 11 / 20 | 14 / 20 | 60 / 100 | −3 |
The corrected native path matches the BF16 decisions
Run the already-trained 615-position patient-01 cartridge through real vLLM generation. The native arm stores its values as FP8 E4M3 and dequantizes them inside the fused attention kernel; the fake arms reconstruct FP8-rounded values in BF16 containers before serving. Score the same 20 LongHealth multiple-choice questions with deterministic greedy decoding.
| patient-01 arm | storage / execution | LongHealth | exact sequence vs fixed native | same correct / incorrect decision |
|---|---|---|---|---|
| BF16 cartridge | BF16 K / BF16 V injection | 10 / 20 | 12 / 20 | 20 / 20 |
| Fake K16/V8, per-tensor scale | FP8-rounded V reconstructed in BF16 | 9 / 20 | 11 / 20 | 19 / 20 |
| Fake K16/V8, unit scale | FP8-rounded V reconstructed in BF16 | 11 / 20 | 15 / 20 | 19 / 20 |
| Fake K8/V8, per-tensor scale | FP8-rounded K and V reconstructed in BF16 | 10 / 20 | 3 / 20 | 16 / 20 |
| Native fused K16/V8 | BF16 K / physical FP8 V | 10 / 20 | reference | reference |
What counts as an acceptable accuracy tradeoff?
There is no universal point budget. Set the limit per workload, using enough samples to resolve the difference, and require the deployment benefit to justify it. Two public industry examples put the numbers in perspective.
MLPerf uses 99% and 99.9% accuracy gates
MLPerf Inference sets workload-specific floors. Its Mixtral question-answering, math, and code workload—which includes GSM8K—requires 99% of the FP16 reference. Several other workloads provide both 99% and stricter 99.9% tracks.
Borrow that rule only as an illustrative bar for these Qwen3-8B results; this is not an MLPerf submission.
Scale spans 98% to 100.25% of each BF16 score. On this borrowed yardstick, K16/V8 clears 99% on both benchmarks; symmetric K8/V8 misses it on GSM8K.
Meta released a mobile model with larger point drops
Meta's released Llama 3.2 3B SpinQuant model moves MMLU from 63.4 to 62.0 (−1.4 points) and GSM8K from 77.7 to 75.7 (−2.0 points).
| OnePlus 12 result | SpinQuant vs BF16 |
|---|---|
| Decode rate | 2.6× |
| Model file | 60.3% smaller |
| Resident memory | 49.8% lower |
Meta reports that tradeoff for constrained on-device inference. It quantizes weights and activations, so it is a concrete industry precedent, not a directly comparable KV-only result.
What fake quantization does with the other eight bits
The loss study rounded each BF16 value to the nearest FP8-E4M3 representable value, then stored that reconstructed number in a BF16 tensor so the existing model code could consume it. The BF16 container still occupies 16 bits, but the lower precision bits contain no independent information. They are fixed by the exact FP8 value represented in BF16; the conversion does not invent eight bits of new attention data.
The arithmetic uses the reconstructed numerical value. It does not ignore half of a BF16 register, but only the FP8-level information survives the round trip. Therefore fake quantization measures numerical damage and saves no memory. The serving kernel below stores a real eight-bit V payload and converts it while loading into the attention pipeline.
One kernel spans cartridge and live KV
Avoid the original two-attention-call design and its exact LSE merge. Present the shared cartridge and each request's paged live suffix as one logical sequence, then feed both regions into FlashInfer's existing register-resident online softmax.
Bypass page-table division and index loads. Issue FP8 value reads before conversion and shared-memory stores to expose memory-level parallelism.
One attention launch avoids a second output, an LSE tensor, a temporary merge buffer, and a separate merge kernel. This specialization also disables internal split-KV partial outputs.
Use a 64×128 tile with one consumer warpgroup. Give producer and consumer 216 registers each; the final kernel uses 32 bytes of stack.
Retain the 128×96 prefill tile. The custom dtype pair and mixed-source planning cover both prefill and decode.
Two fixes made the native result real
The editable FlashInfer installation on gpu1 still pointed its JIT include and source symlinks at an obsolete July checkout. The benchmark driver came from the fused branch, but runtime compilation consumed headers that predated the cartridge-aware mainloop. Binding the JIT paths to the same 68153db5-based tree makes the compiled kernel match the code under test.
A 615-token boundary splits a warp between cartridge values and live values. The cartridge-value offset used __shfl_sync(FULL_MASK) only inside the cartridge branch, so lanes named by the mask did not all execute the shuffle and the warp could deadlock. Compute both source offsets before the branch; every lane now executes both full-mask shuffles.
Memory behavior
K16/V8 payload for one 632-position Qwen3-8B cartridge, down from 88.875 MiB BF16.
One shared cartridge replaces 64 copies of an 88.875 MiB BF16 prefix in the requests' ordinary caches.
Reported live KV-token capacity rises from 365,200 to 486,928 when the whole cache uses K16/V8.
The implementation is split between FlashInfer and vLLM
FlashInfer owns dtype-specialized attention and the fused mixed-source kernel. vLLM owns cartridge residency, scheduler allocation, exact-length planning, backend dispatch, and the end-to-end benchmark. The odd-boundary repair changes only FlashInfer; the existing vLLM integration runs unchanged once its JIT points at the repaired FlashInfer tree.
Review the repaired FlashInfer branch
8b3608e6 lets custom decode and prefill modules compile distinct key and value dtypes.
68153db5 fuses the dense shared cartridge with paged live KV inside one Hopper FA3 attention pass.
fd1a21de keeps full-mask source-offset shuffles converged across odd cartridge boundaries and adds the 615-token regression shape.
Review the exact-length vLLM branch
1dfe7de2e0 preserves split K and V planes in GPU residency.
c04e7137d1 releases cache blocks owned by the external cartridge.
0ec079cdf1 plans and serves the cartridge and live suffix in one pass.
a7cb3769f5 adds the fresh-process four-layout benchmark and summary.
627eea6875 preserves exact cartridge lengths with one-token FlashInfer pages, including the 615-, 644-, 506-, and 566-position artifacts that do not divide by eight.
Validation gates
| gate | coverage | result |
|---|---|---|
| FlashInfer boundary matrix | prefill/decode × BF16/FP8 live V; aligned token-632/page-8 and odd token-615/page-1 boundaries | 8 / 8 pass |
| Deployed-length matrix | the same four cases at 615, 632, 644, 506, and 566 cartridge positions | 20 / 20 pass |
| vLLM connector, core, fused attention | residency, allocation, routing, planning, backend dispatch | 81 / 81 pass |
| Static checks | Ruff, formatter, git diff whitespace | pass |
| Real vLLM generation | patient-01, 20 deterministic LongHealth questions; compare native K16/V8 with BF16 injection | 10 / 20 each; decisions match 20 / 20 |
| Native reproducibility | repeat the corrected native patient-01 run from a warm cache | 20 / 20 token sequences match |
The three-checkpoint objective study remains the strongest measured quantization signal. The patient-01 generation result validates native execution against BF16 on one 20-question slice; a corrected five-patient native rerun remains necessary for a full generation-quality estimate.
Keep the result inside its tested envelope
The fused specialization targets SM90 / FlashInfer FA3. An AMD W7900 can reproduce the fake-quant numerical study, but it cannot compile or execute this CUDA/Hopper kernel. Ampere and ROCm performance remains unmeasured.
The current vLLM path excludes CUDA graphs. Measure graph-compatible integration separately.
Routing supports a singleton shared cartridge per request group. Multi-cartridge composition is not implemented in this specialization.
One-token FlashInfer pages preserve non-divisible lengths. The tested 615-, 632-, 644-, 506-, and 566-position cartridges all pass prefill and decode with BF16 and FP8 live values.
Data-context parallelism and speculative decoding are outside the current contract.
The packed cartridge path supports BF16 keys and FP8-E4M3 values. Other value formats need their own numerical and performance gates.