Compress the shared cartridge.
Keep the keys.
MMLU (Massive Multitask Language Understanding) samples many academic subjects to test broad knowledge and reasoning; GSM8K (Grade School Math 8K) uses math word problems to test multi-step reasoning. Together they are useful, complementary checks of general model quality, although these benchmark runs do not by themselves certify cartridge quality.
Cartridges expose a quantization penalty that those ordinary checks miss. In ordinary Qwen3-8B inference—no cartridge, unchanged model weights, and only the runtime KV cache changed—symmetric FP8 raises the measured MMLU score from 77.360% to 77.458%: a tiny +0.099 percentage-point improvement in this run. It lowers GSM8K from 93.0% to 91.5% (−1.5 points). Apply the same symmetric FP8 fake quantization after training to an existing cartridge, however, and its evaluation loss is 0.873–1.303% higher than the same cartridge in BF16. Quantizing the keys causes almost all of this penalty. K16/V8 keeps keys in BF16 and quantizes only values, holding the loss increase to 0.029–0.038% while reducing the raw KV payload by 25%.
MMLU moves slightly upward. +0.099 points is roughly one additional correct answer per 1,000 questions if interpreted literally; one run does not establish that FP8 improves the model. GSM8K records three fewer correct answers out of 200.
The ordinary benchmarks above hide this cartridge-specific cost. Symmetric K8/V8 raises loss on every cartridge and reaches +1.303%; K8 alone lands in nearly the same range, so keys cause almost all the damage.
K16/V8 saves 22.219 MiB (25%) per cartridge. Green marks the selected layout that retains BF16 keys. Red marks layouts that quantize keys and incur the loss penalty shown in panel 2.
This is post-training quantization. Start with an existing BF16 cartridge, leave its keys unchanged, and convert only its values to FP8 E4M3. The fake-quant experiment measures the numerical effect of that conversion on already-trained cartridges. The fused serving path stores the values as actual eight-bit data and converts them inside attention. Cartridge training is unchanged.
The deployment path is real, not fake quantization: a shared 632-position K16/V8 cartridge stays outside every request's live cache, and one FlashInfer FA3 kernel attends over the cartridge and live suffix with one online-softmax state. With a BF16 live cache, it improves both measured throughput and TTFT against ordinary BF16 cartridge injection at batches 1, 16, and 64.
The fused path wins against a regular BF16 cartridge
Qwen3-8B on one H100 SXM 80 GB, exact 632-position patient-02 cartridge, vLLM eager mode, FlashInfer FA3, 32 generated tokens per request, two exact-shape warmups, and five measured repetitions. Report medians. Configure every engine with max_num_seqs=64 and max_num_batched_tokens=16384.
64.44 → 64.58 output tok/s. TTFT p50 improves 32.72 → 32.34 ms (−1.17%).
887.12 → 920.98 output tok/s. TTFT p50 improves 92.65 → 69.37 ms (−25.13%).
2,340.33 → 2,400.47 output tok/s. TTFT p50 improves 323.76 → 220.63 ms (−31.85%).
| batch | ordinary BF16 cartridge | shared K16/V8 cartridge | tok/s delta | ordinary TTFT p50 | shared TTFT p50 | TTFT delta |
|---|---|---|---|---|---|---|
| 1 | 64.44 tok/s | 64.58 tok/s | +0.22% | 32.72 ms | 32.34 ms | −1.17% |
| 16 | 887.12 tok/s | 920.98 tok/s | +3.82% | 92.65 ms | 69.37 ms | −25.13% |
| 64 | 2,340.33 tok/s | 2,400.47 tok/s | +2.57% | 323.76 ms | 220.63 ms | −31.85% |
When the model already uses K16/V8
Compare the shared cartridge path with the ordinary globally asymmetric K16/V8 engine, not with BF16. Sharing still cuts TTFT and avoids per-request cartridge copies. The custom mixed-source kernel does not beat the stock global path at batch-64 sustained decode.
| batch | ordinary global K16/V8 | shared cart + global K16/V8 | tok/s delta | ordinary TTFT p50 | shared TTFT p50 | TTFT delta |
|---|---|---|---|---|---|---|
| 1 | 62.61 tok/s | 62.21 tok/s | −0.63% | 34.13 ms | 32.93 ms | −3.53% |
| 16 | 883.24 tok/s | 884.16 tok/s | +0.10% | 84.08 ms | 70.28 ms | −16.41% |
| 64 | 2,456.57 tok/s | 2,196.87 tok/s | −10.57% | 289.89 ms | 223.46 ms | −22.92% |
Keys carry the cartridge quantization loss
Qwen3-8B has 36 layers, eight KV heads, and head dimension 128. The patient-02 cartridge has 632 positions: one frozen attention sink and 631 trainable positions. Report raw KV payloads without scale or serialization metadata.
| layout | bytes / token | full 12,628-token KV | 632-position cartridge | capacity | in-sample eval loss increase |
|---|---|---|---|---|---|
| BF16 K16/V16 | 144 KiB | 1.734 GiB | 88.875 MiB | 1.00× | baseline |
| Asymmetric K16/V8 | 108 KiB | 1.301 GiB | 66.656 MiB | 1.333× | +0.031%, +0.029%, +0.038% |
| Reverse asym. K8/V16 | 108 KiB | 1.301 GiB | 66.656 MiB | 1.333× | +0.837%, +1.271%, +1.140% |
| Symmetric K8/V8 | 72 KiB | 0.867 GiB | 44.438 MiB | 2.00× | +0.873%, +1.303%, +1.177% |
Mean relative objective increase across the three separately trained cartridge checkpoints.
Quantizing keys causes essentially all of the measured cartridge loss.
Symmetric FP8 gains 2× payload capacity but misses the cartridge quality knee.
What the three loss values mean
| trained cartridge | BF16 eval loss | K16/V8 increase | K8/V8 eval loss | K8/V8 increase |
|---|---|---|---|---|
| CAS cartridge run 1 | 0.016542 | +0.031% | 0.016687 | +0.873% |
| Half-learning-rate ablation | 0.016799 | +0.029% | 0.017018 | +1.303% |
| CAS cartridge run 2 | 0.016932 | +0.038% | 0.017131 | +1.177% |
These are three separately trained cartridge artifacts, not three samples from one inference run. The half-learning-rate arm is an ablation; run 2 is a second CAS-regime cartridge. All loss values in this table are post-training, in-sample evaluations.
Why ordinary inference did not warn us
| measurement | BF16 K16/V16 | asymmetric K16/V8 | symmetric K8/V8 |
|---|---|---|---|
| Regular inference MMLU | 77.360% | 77.393% (+0.033 pp) | 77.458% (+0.099 pp) |
| Regular inference GSM8K (n=200) | 93.0% | 92.5% (−0.5 pp) | 91.5% (−1.5 pp) |
| In-sample cartridge eval loss | baseline | +0.029% to +0.038% | +0.873% to +1.303% |
The ordinary full-model benchmarks do not reveal the cartridge-specific penalty. In this run, symmetric FP8 moves the measured MMLU score upward by 0.099 points—too little evidence to conclude that quantization improves the model—while GSM8K loses three correct answers out of 200. The paired cartridge evaluation consistently detects the cost of quantizing keys. Treat a cartridge as its own quantization qualification workload; ordinary generation scores alone do not establish cartridge safety.
What counts as an acceptable accuracy tradeoff?
There is no universal point budget. Set the limit per workload, using enough samples to resolve the difference, and require the deployment benefit to justify it. Two public industry examples put the numbers in perspective.
MLPerf uses 99% and 99.9% accuracy gates
MLPerf Inference sets workload-specific floors. Its Mixtral question-answering, math, and code workload—which includes GSM8K—requires 99% of the FP16 reference. Several other workloads provide both 99% and stricter 99.9% tracks.
Borrow that rule only as an illustrative bar for these Qwen3-8B results; this is not an MLPerf submission.
Scale spans 98% to 100.25% of each BF16 score. On this borrowed yardstick, K16/V8 clears 99% on both benchmarks; symmetric K8/V8 misses it on GSM8K.
Meta released a mobile model with larger point drops
Meta's released Llama 3.2 3B SpinQuant model moves MMLU from 63.4 to 62.0 (−1.4 points) and GSM8K from 77.7 to 75.7 (−2.0 points).
| OnePlus 12 result | SpinQuant vs BF16 |
|---|---|
| Decode rate | 2.6× |
| Model file | 60.3% smaller |
| Resident memory | 49.8% lower |
Meta reports that tradeoff for constrained on-device inference. It quantizes weights and activations, so it is a concrete industry precedent, not a directly comparable KV-only result.
What fake quantization does with the other eight bits
The loss study rounded each BF16 value to the nearest FP8-E4M3 representable value, then stored that reconstructed number in a BF16 tensor so the existing model code could consume it. The BF16 container still occupies 16 bits, but the lower precision bits contain no independent information. They are fixed by the exact FP8 value represented in BF16; the conversion does not invent eight bits of new attention data.
The arithmetic uses the reconstructed numerical value. It does not ignore half of a BF16 register, but only the FP8-level information survives the round trip. Therefore fake quantization measures numerical damage and saves no memory. The serving kernel below stores a real eight-bit V payload and converts it while loading into the attention pipeline.
One kernel spans cartridge and live KV
Avoid the original two-attention-call design and its exact LSE merge. Present the shared cartridge and each request's paged live suffix as one logical sequence, then feed both regions into FlashInfer's existing register-resident online softmax.
Bypass page-table division and index loads. Issue FP8 value reads before conversion and shared-memory stores to expose memory-level parallelism.
Launch attention once. Do not materialize a second output, an LSE tensor, a temporary merge buffer, or a separate merge kernel. Disable internal split-KV partial outputs for this specialization.
Use a 64×128 tile with one consumer warpgroup. Give producer and consumer 216 registers each; the final kernel uses 32 bytes of stack.
Retain the 128×96 prefill tile. The custom dtype pair and mixed-source planning cover both prefill and decode.
Memory behavior
K16/V8 payload for one 632-position Qwen3-8B cartridge, down from 88.875 MiB BF16.
Do not duplicate an 88.875 MiB BF16 prefix into every request's ordinary cache. Keep one shared cartridge instead.
Reported live KV-token capacity rises from 365,200 to 486,928 when the whole cache uses K16/V8.
The implementation is split between FlashInfer and vLLM
Use the same branch name in both public forks: 20260911-fused-asym-cartridge. FlashInfer owns dtype-specialized attention and the fused mixed-source kernel. vLLM owns cartridge residency, scheduler allocation, request planning, backend dispatch, and the end-to-end benchmark.
Review the FlashInfer branch
8b3608e6 lets custom decode and prefill modules compile distinct key and value dtypes.
68153db5 fuses the dense shared cartridge with paged live KV inside one Hopper FA3 attention pass.
Review the vLLM branch
1dfe7de2e0 preserves split K and V planes in GPU residency.
c04e7137d1 releases cache blocks owned by the external cartridge.
0ec079cdf1 plans and serves the cartridge and live suffix in one pass.
a7cb3769f5 adds the fresh-process four-layout benchmark and summary.
Validation gates
| gate | coverage | result |
|---|---|---|
| FlashInfer numerical matrix | prefill/decode × BF16/FP8 live V; exact token-632 boundary; BF16 PyTorch reference at rtol=atol=3e-2 | 4 / 4 pass |
| vLLM connector, core, fused attention | residency, allocation, routing, planning, backend dispatch | 81 / 81 pass |
| Static checks | Ruff, formatter, git diff whitespace | pass |
| Real generation smoke | four LongHealth questions: BF16 3/4, K16/V8 2/4 | smoke only |
Do not use the four-question smoke as a quality estimate. Use the three-checkpoint objective result above for the measured quantization signal and run a larger generation evaluation before a production quality claim.
Keep the result inside its tested envelope
The fused specialization targets SM90 / FlashInfer FA3. Do not infer Ampere or ROCm performance.
The current vLLM path excludes CUDA graphs. Measure graph-compatible integration separately.
Routing supports a singleton shared cartridge per request group. Multi-cartridge composition is not implemented in this specialization.
The tested 632-position cartridge aligns exactly to eight-token pages. Reject unsupported boundary layouts instead of silently falling back.
Data-context parallelism and speculative decoding are outside the current contract.
The packed cartridge path supports BF16 keys and FP8-E4M3 values. Other value formats need their own numerical and performance gates.