Asymmetric KV
Cache Quantization

Precision and scaling · two Qwen checkpoints

Precise keys matter for Qwen2.5. Qwen3 has more options.

Asymmetric KV is useful when compressing keys damages quality. On Qwen2.5-7B-Instruct, keeping keys precise avoids a severe math and retrieval failure under both tested scaling policies. On Qwen3-8B, symmetric FP8 remains a credible alternative: it saves more memory and passes the reference test for average math accuracy. These results do not establish a general asymmetric quality or speed advantage across models.

Why the quality verdict depends on the objective: answer preservation bounds how many previously correct answers become wrong (at most 2% of all questions). Average accuracy allows gains to offset losses and must rule out a drop larger than one percentage point. Both include prediction-loss and retrieval checks. These are different requirements; neither is always stricter. Missing a confidence bound does not by itself prove worse quality.

Coverage: seven cache/scaling configurations ran on each of two checkpoints from the Qwen family. Only Qwen3-8B received the new actual-worker comparison. These panels do not establish a ranking across model families or a serving-speed winner. Exact coverage and remaining comparisons · Worker counts and memory · Reference results →

Measured inference value

Cache savings and speed are separate results.

The key/value (KV) cache stores information from previous tokens. Keys determine what attention selects; values supply the selected information. K16/V8 keeps keys at 16 bits and stores values at 8 bits. Model weights are separate: INT4 weights do not imply an INT4 KV cache.

HTTP serving: a smaller gain, with limits

Dense Qwen3-8B passes the HTTP worker’s quality checks at 1K and 8K input. At 1K with 32 requests in flight, throughput improves 1.01% versus native FlashInfer. A gain versus native FlashAttention 3 (FA3) is not established. Request concurrency is not the GPU batch size.

At 8K, observed throughput increases do not qualify as a confirmed serving gain: 12 requests failed across the three arms. At the shared 2.9 requests/s offered load, p95 request latency is 4.34% higher than FlashInfer and 12.65% higher than FA3. Higher latency is slower; p95 is the 95th percentile.

HTTP tests use equal cache byte budgets, giving K16/V8 more token slots. They do not demonstrate a 25% reduction in whole-GPU memory or a general latency improvement. Throughput, time to first token, time per output token, failures and reproduction →

Choosing the precision · Qwen3.8 and Gemma 4

Symmetric FP8 is the stronger memory candidate for Qwen3.8.

Both K16/V8 and K8/V8 pass Qwen3.8-27B’s quality checks, while K8/V8 saves twice as many state bytes. Gemma 4 26B-A4B passes math and retrieval with both formats, but neither meets every prediction-loss limit. These are quality and physical-storage results; serving speed was not tested.

Measured after 8,192 input + 128 continuation tokens, with original BF16 weights and compute. Here K16 means BF16; the optimized Qwen3 recipe uses FP16. Each model’s bars are scaled to its own native state size. Sizes include FP32 scales; weights, temporary reconstruction and allocator reservations are separate.

Every arm ran all 1,319 GSM8K math questions, retrieval at 1K and 8K, and prediction-loss checks. The direct math comparison does not establish an accuracy advantage for either compressed format. Passing these tests is not a guarantee for every task.

Full quality results, state breakdown and reproduction →

Optimized stack compatibility

Start with the qualified Qwen3 recipe.

A quality pass covers the tested prediction-loss, retrieval and full GSM8K math checks. It does not establish universal accuracy.

The Qwen3.8-27B and Gemma 4 26B-A4B reference results above do not qualify these models for the fused stack. Unsupported layouts are not measured quality failures. Quality counts, limits and model details →

How it works

Keep the compressed cache through decode.

The optimized vLLM route also handles fresh whole prompts when other requests are decoding. The measured 8K batch improvement comes from the complete request path, including prefill routing; it does not establish a faster decode kernel. Routing diagram and fallback details →

Install and run

Install the tested batch recipe.

Use Linux x86-64, a CUDA 12.9 development environment, at least 100 GB system RAM and 100 GB free disk. Install Git, a C/C++ compiler and uv 0.8.17 using the complete setup instructions. The installer selects exact fork commits, checks the base wheel and records a package lock. No LMCache or private inputs are required.

git clone https://github.com/mcgrof/knlp.git
cd knlp
git switch main
export FLASHINFER_WORKSPACE_BASE="$PWD/stack/jit"
uv run --python 3.12.11 --no-project \
  -m tools.kv.native.october_setup "$PWD/stack" --recipe current
export KNLP_KV_FIXTURES=/path/to/experiment-inputs
stack/env/bin/python -m tools.kv.native.current download \
  --format awq --output awq-download.json
stack/env/bin/python -m tools.kv.native.current smoke \
  --install "$PWD/stack" --format awq --output awq-smoke

Fresh H100 installation passed for AWQ and dense FP16. Both selections passed sequentially on October 19, 2026, using installer 5a3d9d74. See the receipt and setup/check times.

The check loads the supplied calibrated scales before graph capture, generates a finite answer and verifies mixed prefill/decode admission, physical storage, changing graph inputs and memory accounting. A successful run writes receipt.json with "passed": true. Use --format fp16 in both commands for the dense selection, after the first process exits.

PASS: physical K16/V8; calibrated scales; mixed fresh-prefill route; changing decode graphs; clean memory; finite deterministic generation

This tries and validates the feature. To reproduce the numbers, use the frozen inputs, measured harness revisions and full protocol. The quick-start check is not a benchmark. Public model downloads need no token; optional authentication uses hf auth login.

Integration status

FlashInfer first, then vLLM integration.

vLLM upstreaming waits for the FlashInfer asymmetric K/V capabilities it uses to merge. The tested recipe is available now through pinned public forks. LMCache storage work can proceed in parallel. Status checked October 19, 2026.

See the itemized PR series, seven fixes, branch links and dependencies →

LMCache is optional for the current resident-GPU batch recipe. Full FP8 byte-through serving requires a matching vLLM/FlashInfer producer; the storage code can be reviewed independently.

Inspect and reproduce

The evidence, one level deeper.

Reference model comparison: Qwen3.8-27B and Gemma 4 26B-A4B →