Protect the keys.
Quantize the values.
Do not choose one KV-cache dtype for every model. Symmetric FP8 is a useful capacity tier when the model's keys tolerate it. It can fail catastrophically on models with fragile, biased key geometry. For those models, keep keys in BF16 and store values in FP8: asymmetric K16/V8 preserves measured quality while reducing KV bytes by 25%.
Pre-bias confirmed the diagnosis: removing the key bias restores long-context retrieval while keeping K8/V8 storage. It did not become the serving fix. The final equal-memory gate showed a decisive throughput and latency regression, so the pre-bias kernel-optimization line is permanently closed and countersigned (2026-09-01). Pre-bias remains a documented capacity option (1.50× the measured K16/V8 token capacity), never the default; K16/V8 remains the default for fragile-key models.
Deploy a policy, not a universal dtype
Capacity is free only after quality and execution support pass. The storage ratios below are exact for conventional caches with equally sized K and V tensors; compressed-latent attention needs its own accounting.
| KV tier | bytes per K/V element pair | capacity vs BF16 | quality rule | deployment status |
|---|---|---|---|---|
| BF16 K / BF16 V | 4 | 1.00× | reference | fallback |
| BF16 K / FP8 V (K16/V8) | 3 | 1.333× | measured quality-preserving knee | fragile-key default |
| FP8 K / FP8 V (K8/V8) | 2 | 2.00× | requires model qualification | robust-key tier |
| pre-biased FP8 K / FP8 V | 2 | 2.00× | retrieval repaired | not the serving default |
| sub-8-bit frontier | varies | >2.00× possible | keys dominate error; values are safer | research tiers |
Use symmetric FP8 only after long-context retrieval and generation probes show that key quantization is admissible for the model and serving stack.
Use K16/V8 to retain the value-side savings without exposing the model to destructive key quantization.
Do not infer safety from weight precision, model family labels, perplexity alone, or a kernel benchmark. Run the model-level gate.
The R&D chain is now coherent
Each page answers a different question. Read them in this order instead of treating the results as independent threads.
Keep the pre-bias finding; stop the kernel line
Pre-bias answered the scientific question and failed the engineering decision. Those are compatible conclusions.
Retrieval is repaired
Pre-bias restores needle retrieval at 8K, 16K, and 30,720 tokens where ordinary symmetric K8/V8 scores 0.00 on the tested fragile-key model (Qwen2.5-7B; the biased-key Qwen2/Qwen2.5 family).
Served geometry passes
The clean kernel passes all eight mutation cases at the served geometry. Injected live defects fail as intended.
Performance fails
Pre-bias at concurrency 48 delivers 0.654× the completed-request rate of K16/V8 at concurrency 32, with 2.056× p95 request latency (Qwen2.5-7B on an H100, 30,720-token inputs with 128-token outputs; result commit 85fa7b2d, countersigned in c82c5db9; serving stack: vLLM fork 8a1714108, FlashInfer fork 6dfdc833).
| decision metric | required | measured | verdict |
|---|---|---|---|
| completed requests per second | at least 1.20× K16/V8 | 0.654× | fail |
| p95 request latency | at most 1.25× K16/V8 | 2.056× | fail |
| served mutation suite | all clean cases pass; defects fail | meets gate | pass |
| long-context retrieval | repair ordinary K8/V8 collapse | 0.00 → 1.00 | pass |
Gate provenance: result commit 85fa7b2d, countersigned c82c5db9 (2026-09-01); serving stack: vLLM fork 8a1714108, FlashInfer fork 6dfdc833.
Pre-bias is evidence that key bias causes the failure and that a bias-aware representation can recover quality at symmetric-FP8 capacity.
Do not spend another tuning cycle on this kernel. The pre-bias kernel-optimization line is permanently closed and countersigned (2026-09-01). New hardware, a different kernel family, or a materially different representation would open a new line, not reopen this one.
Separate storage, quality, and kernel speed
A dtype does not have one universal speed. Kernel family, batch, layout, transforms, and launch policy decide whether reduced bytes become useful throughput.
Ordinary K8/V8 is faster than K16/V8 at the measured bandwidth-bound shapes within the same audited kernel family (result commit f75960a6).
The same K8/V8 path loses at batch 1. Context length alone does not predict the sign of the result.
Numerical repair does not erase transformation and scheduling cost. The full serving gate, not a microbenchmark, sets the default.
Preserve exact kernel, build, layout, scale, transform, and launch provenance with every performance result. Do not compare storage formats across unrelated kernel families as if dtype were the only variable.
Carry the policy into capacity planning
No additional public results page is needed for the two planning tools. They are operational consequences of this policy and belong here, next to the decision they implement.
aiconfigurator: K16/V8 byte accounting v2
Represents key and value widths separately, computes the exact 3-byte K/V pair for conventional attention, and exposes K16/V8 as a capacity-only mode. It rejects compressed-latent caches and refuses to invent latency without measured mixed-dtype kernel rows.
Review the v2 branch →InferenceX: requested/effective dtype provenance v2
Records requested key and value dtypes separately, identifies symmetric and asymmetric representations, and keeps effective dtype unknown unless the engine attests it. It stores an argv fingerprint, never the raw server command line.
Review the v2 branch →What the research established
The assessment is closed; production R&D is not
The pre-bias decision and the high-level quantization policy are complete. The remaining work is integration, broader validation, and operational proof.
Land K16/V8 cache allocation, writers, decode kernels, serialization, and API plumbing in the serving projects that own them.
Have each engine emit the resolved key/value representation after configuration, checkpoint defaults, and backend selection. Requested flags are not proof of runtime state.
Compare planner estimates with allocator telemetry across block sizes, prefix caching, sliding windows, offload, speculative decoding, tensor parallelism, and disaggregated serving.
Extend model, context, workload, hardware, and kernel-family coverage. Treat new accelerators and compressed-latent attention as new measurement domains.
Test save/restore, transfer, eviction, swapping, prefix reuse, LMCache codecs, and mixed-version compatibility with asymmetric K/V dtypes.
Turn key-fragility probes, long-context retrieval, generation quality, capacity, and serving performance into a repeatable admission test for each model and release.