Historical reference. This page preserves the research record as of October 19. For the supported recipe and current limits, see Asymmetric KV Cache Quantization. The completed dense 1K check supersedes the proposed test below.

Memory-Traffic Saturation in Autoregressive Transformer Decode

Smaller KV cache.
A measured serving gain.

Asymmetric KV caching improves complete-request throughput with both packed INT4 AWQ and dense FP16 Qwen3-8B weights. Keeping keys in FP16 and values in FP8 gives 3.45% and 3.55% gains, respectively, over matched native FP16 KV controls at 8K input and batch 32. Each weight format passes its own production, task-quality and speed gates.

October 2026 · current finding Better prefill routing makes the smaller cache faster at the same batch. The gain transfers across two weight formats of one model architecture. BF16, other models and the older 1K workload remain separate tests.
8,192 INPUT TOKENS · BATCH 32 · 128 OUTPUTS 1 × H100 SXM 80 GB QUALITY GATES PASS
Read the current result ↓ Use the measured source pair ↓ Explore capacity and throughput ↓

Updated October 19, 2026 · Results through October 18. Earlier measurements and their original limits remain available.

October 2026 · current serving results

One model, two weight formats. Both beat native FP16 KV.

Packed INT4 AWQ weights
+3.45% throughput
Versus native FP16 K/V, with matched weights, workload and a 48 GiB KV byte pool.
Dense FP16 weights
+3.55% throughput
Versus native FP16 K/V, with matched weights, workload and a 40 GiB KV byte pool.
At equal token slots
25% less KV payload
Real FP16-key / FP8-value storage. This is a KV payload saving; total GPU memory also includes weights and workspaces.
Complete-request throughput, including prefill, scheduling and decode. One H100 SXM 80 GB; 8,192 input tokens, batch 32, 128 forced outputs in each study.
Qwen3-8B weightsKV pool per armVersus native FP16 KVVersus preceding compactQuality
Packed INT4 AWQ48 GiB+3.45%+4.88%NLL, retrieval, full GSM8K pass
Dense FP1640 GiB+3.55%+4.73%NLL, retrieval, full GSM8K pass

Each row is a separate matched three-arm test. Its two comparisons win all ten fresh confirmation blocks and pass the frozen 1% median-gain rule, with Holm-adjusted one-sided sign p = 0.001953 each. These are gains against each row's own controls; they do not compare AWQ throughput directly with dense throughput. Read the dense quality and memory checks.

INT4 describes the weights, not the KV cache. AWQ weights stay packed in every AWQ arm; dense weights stay FP16 in every dense arm. Native KV stores both keys and values in FP16; asymmetric KV stores keys in FP16 and values in FP8 E4M3. Both studies use FP16 compute and native FlashInfer controls. Neither establishes a gain over a BF16 serving recipe.

For AWQ, ten fresh, counterbalanced three-arm confirmation blocks give median candidate/native and candidate/preceding-compact ratios of 1.034483× and 1.048814×. Each comparison wins all ten blocks and passes the predeclared 1% practical-gain rule; each Holm-adjusted one-sided sign p-value is 0.001953. Pilot runs are separate. The three-block batch-8 engineering screen passes its 0.99 threshold at 0.99889× versus preceding compact; it does not establish a new batch-8 gain.

What changes in vLLM's prefill routing?

Compact prefill gathers stored K16/V8 pages into one bounded FP16 workspace for native FA3 prefill. Casting values back to FP16 retains their quantization error. Previously, only the first eight of 32 prompts used this route; four later admissions fell back because their batches also contained decoding requests. The new vLLM route selects the fresh prompts inside each mixed batch and maps their query and cache pages correctly. All 32 prompts now use compact prefill. Decode still reads the compressed cache.

Fresh prefill gets a fast path inside a mixed batch

Existing vLLM batch orderDecode requests first · prefill requests after them
Decode prefix · unchangedPaged mixed FA2 decodeReads persistent FP16 keys and FP8 values
Decode output sliceOriginal query positions
Entire prefill suffixAll prompts fresh and whole, with a supported recipe?Check lengths, page mapping and workspace capacity
Extended routeRebase query and page offsetsSelect only the fresh-prefill suffix
Gather K16 / V8 pagesShared 256 MiB FP16 payload staging plus wrapper scratch; cast V8 → FP16
Native FA3 prefillFP16 Q, K and V inputs
Existing fallbackMixed-type prefillFor supported batches outside the compact fast path
Prefill output sliceOriginal query positions
Original batch output tensorEach path writes its own slice; request order stays unchanged
Logical routing, not parallel execution: the shared workspace is reused across layers on one worker's serial CUDA stream. Prefill runs before decode. The persistent cache stays K16/V8; existing value scales still apply in attention. Casting FP8 values to FP16 does not recover lost precision.

The measured route uses VLLM_COMPACT_PREFILL=1 on SM90, FP16 query/key, E4M3 FP8 value, NHD layout, page size 16, 32 query heads, eight KV heads and head dimension 128. Whole prompts have no cached prefix; prefix caching and chunked prefill are off. Every prefill prompt must qualify and have at most 8,192 tokens; otherwise the suffix takes the existing prefill path. The shared payload holds at most 4,096 pages. Direct compact allocations also include 32 MiB float scratch and 16 KiB indices; FlashInfer's internal wrapper buffers are additional. This is a single-worker, causal-attention recipe, without sliding windows, attention sinks or soft caps. Exact subset mapping · Shared workspace and gather.

The gain removes a serving overhead that previously hid the benefit of asymmetric storage. It supports a better complete-request recipe at this workload. The vLLM route changes; the FlashInfer revision stays the same in all three arms. The result does not show a faster decode kernel, or that adding more resident sequences yields the same percentage gain. The saturation model explains why capacity and fixed-batch speed must be measured separately.

AWQ quality and memory qualify the result

Qwen3-8B-AWQ, 8K input, batch 32. Quality comparisons use both native and preceding compact controls.
CheckCurrent evidence
Production routingAll 360 same-dispatch checks pass across 36 layers, five admission cohorts and two prompt rotations; fixed fault injections are detected.
Task qualityFresh NLL on 32 documents, 100 retrieval cases and all 1,319 GSM8K questions pass the frozen preservation gates.
GSM8KCandidate 1,201/1,319; native 1,199; preceding compact 1,201. Gross-loss upper bounds are 1.927% versus native and 0.227% versus preceding compact, within the 2% limit.
Actual storageK16/V8 persistent tensors, one shared 256 MiB payload workspace, no full FP16 value shadow. Clean allocator peak: 63.23 GiB; driver observations every 500 ms reach 63.98 GiB.

The fixed byte pool can use saved bytes for more token slots. This test does not measure a new maximum resident batch, and the driver samples are not a continuous peak measurement. Earlier 58/59/44 capacity endpoints belong to the separately measured preceding compact, mixed and native recipes.

Numerical contract and full measurement data

Routing qualification compares the accepted compact operation with an independent raw-page mapping and the same batched dispatch. It does not establish exact ideal-FP32 attention. The unchanged ideal-reference diagnostic retains 27 output violations among 3,538,944 fixed sampled elements and zero sampled log-normalizer violations. The earlier failed strict-reference attempt remains in the record. These correlated samples are not a model-wide error-rate estimate.

Matched component profiles separate device work, gathering and host launch cost. They explain the route change; the speed decision uses the complete-request trials. All confirmation ratios, quality bounds and source pins · Full method and component measurements.

A separate FlashInfer fix: D64 correctness

The repaired paged FA2 path for 64-element heads passes its declared numerical fixtures. On the original failing fixture, maximum error falls from 0.0467 to 0.0000278 against the unchanged 0.005 limit. This repair is not part of either measured speed stack. The new full-model check unblocks production but fails the frozen retrieval-quality rules. Throughput and contiguous/ragged paths remain unqualified.

Scope across models
The earlier standardized 1K/batch-32 study has two quality-eligible models and no confirmed throughput gain; five models fail a quality gate and one fails a production check. Those results use their original sources and workload. The two new 8K results do not replace that matrix or qualify the BF16 and symmetric-AWQ recipes that failed quality. Read the complete matrix and controls.
Dense transfer · measured October 18

The routing gain transfers to dense FP16 weights

Dense Qwen3-8B also passes at 8K input and batch 32. Ten fresh confirmation blocks give median complete-request throughput gains of 3.55% over native FP16 K/V and 4.73% over preceding compact. Each comparison wins all ten blocks, with Holm-adjusted one-sided sign p = 0.001953. All arms load actual dense FP16 weights and use the same 40 GiB KV byte pool. Native uses FlashInfer FP16 K/V; the routing change uses the same FlashInfer revision.

Separate dense FP16 transfer: Qwen3-8B, 8,192 input tokens, batch 32, 128 forced outputs, one H100 SXM 80 GB.
CheckMeasured outcome
Production and graphsAll 360 routing-equivalence checks pass across 36 layers, five admission cohorts and two rotations. Four fixed faults are detected; actual decode-graph inputs change.
Task qualityNLL on 32 documents, 100 retrieval cases and all 1,319 GSM8K questions pass against both matched controls. Retrieval is 96/100; GSM8K is 1,178/1,319, versus native 1,177 and preceding compact 1,178.
Separate batch-8 guardThree engineering blocks give median candidate/preceding compact 1.00207, above 0.99. This does not establish statistical noninferiority or a new batch-8 gain.
Physical storageNative: 291,264 KV token slots per layer; both asymmetric arms: 388,352. Each has 42,948,624,384 deduplicated K/V backing bytes. The fixed pool has more slots; total GPU allocation does not fall by 25%.
Clean memoryOne 256 MiB compact payload, separate scratch/index buffers, no full FP16 value shadow. Allocator peak 64.76 GiB; 19 driver observations at 500 ms reach 65.51 GiB. No capacity sweep was run.

Matched profiles show the three seven-fresh mixed admissions falling from about 497–502 ms to 307–309 ms for preceding versus candidate gather plus prefill. These are device-operation duration sums, with correlated host launch cost and residual work reported separately. The separate complete-request trials establish the serving gain.

The numerical contract remains routing equivalence to the accepted operation. The unchanged ideal-FP32 diagnostic still fails 9 of 360 checks for both candidate and trusted operation. Full-fallback output and log-normalizer checks pass all 360. This result does not establish ideal arithmetic, BF16 support or new maximum resident-batch endpoints.

Exact sources, quality rules, memory accounting and component measurements · All dense confirmation ratios and sanitized measurements.

D64 repair: production passes, retrieval blocks qualification

The repaired paged path passes Qwen2.5-0.5B's original production checks at 1K input, batch 32 and an 8 GiB pool, including physical bytes, exhaustive writers and changing model graphs. It remains quality-ineligible. NLL passes, but retrieval scores 41/100 for native and 43/100 for repaired. The native control misses its 50% floor; two gross losses give a 6.16% upper bound, above the 5% limit. This fails to establish preservation; it does not demonstrate quality harm.

The frozen pre-GSM stop rule gates off GSM8K, and no comparative timing follows. A separate production-only validation retains exact loaded binaries by hash; the recorded negative quality panel is never repeated or replaced. The original historical failure row remains unchanged. D64 measurements and source identities · Full method and limits.

What should be tested next?

The next dense bridge is the original 1K/batch-32 workload under a separately frozen plan. Compare this routing implementation with preceding compact and the native control at matched inputs, weights and KV byte pool. Include the original matrix's native FLASH_ATTN/FA3 backend when testing whether its result changes; the current 8K native control uses FlashInfer. Require production and task-quality gates before fresh, counterbalanced complete-request timing. Report speed separately from physical KV capacity.

That test can show whether the 8K gain survives at the older 1K workload. It cannot be assumed from these results. BF16, other model geometries, wider batch/context ranges and a combined routing-plus-D64 stack remain separate questions. The D64 model's weak native retrieval control needs investigation before a new quality study; keep this failed gate in the record. The earlier eight-model matrix and dated video retain their original limits.

01 · The Saturation Law

More simultaneous work. A smaller marginal gain.

The law fits aggregate decode throughput as active batch grows. At a fixed context length, throughput rises quickly at first and then flattens. A larger cache can admit more sequences, but it cannot by itself make the GPU process those sequences proportionally faster.

S(B) = Smax · Bγ / (B½γ + Bγ)
B
Active sequences in a decode batch. Requested client concurrency may include queued requests and is a different measurement.
S(B)
Total output tokens per second across that batch, rather than the token rate seen by one user.
Smax
The fitted throughput ceiling for the configuration and context being measured.
B½
The batch that reaches half the fitted ceiling. It describes where diminishing returns become important.
γ
The shape exponent. Fit it from measurements; γ = 1 gives the simple saturating curve illustrated below.

Before the plateau

More active sequences can amortize shared work, including weight access, and improve GPU utilization. Relieving a KV-capacity limit here can unlock substantial throughput.

Near the plateau

Each new sequence still adds KV traffic and computation. More sequences share nearly the same total token rate, so additional capacity buys little throughput and can worsen per-user latency.

This is an empirical Hill fit. It describes a response curve; a good fit does not identify a unique bottleneck or establish that architecture is irrelevant. Model, GPU, context, precision, kernel, parallelism, and scheduling all define the measurement. Fit each relevant context separately. If the measured batches never approach the plateau, the extrapolated ceiling is poorly determined.

What does extra KV capacity buy?

Illustration: γ = 1, unchanged throughput curve, KV-limited batch, enough demand. These are calculated values, not benchmark observations.

For γ = 1, 33⅓% more batch at half the ceiling increases throughput by 14.3%. At 90% of the ceiling the gain is 2.6%.
Starting batchExpanded batch

33.3% more batch → 14.3% more throughput

50.0% → 57.1% of the ceiling. Approximate time per output token rises 16.7%.

The latency estimate uses batch / aggregate throughput for one token per active sequence per decode step. It excludes queueing, prefill, and speculative acceptance. Real serving must measure latency directly.

25% fewer KV bytes can mean 33⅓% more KV capacity. For equally sized K and V tensors, K16/V16 uses four bytes per element pair and K16/V8 uses three. The inverse ratio is 4/3. This is a payload calculation: weights, scale metadata, block fragmentation, workspaces, shared prefixes, and offload buffers still consume memory. Compressed-latent and hybrid caches require their own accounting. None of these ratios establishes numerical quality.

Moving along the curve and changing the curve are different effects. More capacity can increase active batch. A faster attention kernel can raise throughput at the same batch. Reconstruction or transfer work can lower it. Measure fixed-batch performance first, then compare the larger admissible batch under the same quality and latency requirements. Compression may still help at saturation if it reduces the traffic that sets the ceiling; the illustration holds that effect fixed.

The marginal return, expressed mathematically

The local percentage response to batch is d ln S / d ln B = γ · (1 − S/Smax). At γ = 1 and 90% of the ceiling, a small 1% batch increase buys about 0.1% more throughput. For a finite capacity multiplier f at γ = 1, the throughput multiplier is f · (B½ + B) / (B½ + fB).

A useful intuition is step time ≈ a + bB: shared cost a plus per-sequence cost bB. Then aggregate throughput B/(a + bB) has the γ = 1 Hill form. This simplified model explains the shape; it is not a proof that real kernels have only these two costs.

The earlier INT4 speedup fit measures a different quantity

The earlier controlled H100 study fit the same functional form to the speedup ratio of fused INT4 Triton attention against FP16 PyTorch SDPA: 767 decode-mode configurations with B ≥ 4 across 14 models. Its reported parameters were Rmax = 3.75×, B½ = 5.1, γ = 1.32, R² = 0.80. Here the vertical axis is a dimensionless ratio, not aggregate tokens per second.

The fitted curve reaches at least 80% of its asymptotic ratio at B = 16 and at least 95% at B = 64. These coefficients belong to that kernel comparison. They do not predict a 3.75× K16/V8 serving gain, and R² = 0.80 describes fit quality in the observed sample rather than universal predictive accuracy.

Historical fitted INT4 attention ratio: ceiling 3.75×, half-saturation batch 5.1, exponent 1.32.

Only the fitted curve is plotted. No inferred points are labeled as measured observations. See the fused-quantization experiments for the controlled kernel work.

Background and reproduction: decode memory-traffic characterization · characterization runner · serving and saturation workflows.
BACKGROUND · OPTIONAL READING

The ideas behind the measurements

These explainers expand the attention loop, hardware scaling, and statistical interpretation used in the study.

Part 1 — Structural
Autoregressive Decode Bottleneck
Per-step KV reread pressure in dense causal attention. Total reads = G·C₀ + G(G−1)/2. Shows why the loop is the bottleneck before any hardware is involved.
Part 2 — Empirical
KV-Memory Traffic Governs Decode
Decode scaling across GPU classes. Hill-type saturation law, pipeline comparisons, bandwidth plateau behavior. The empirical counterpart to Part 1's structural argument.
Part 3 — Statistical
Spearman ρ — Rank Correlation
The paper uses Spearman rank correlation to test whether architectural features predict KV quantization sensitivity. The tested predictors did not show significant rank correlation in this sample. Absence of significance does not prove absence of an architectural effect.
R&D

Earlier R&D — Fusion Principle Proof (Custom INT4 Triton Kernel)

Earlier custom INT4 Triton experiments compare reconstruction into an intermediate FP16 buffer with dequantization inside the attention loop. They illustrate the cost of extra memory traffic in decode. The current serving result also depends on prefill routing: a bounded shared reconstruction workspace helps prefill, while decode continues to consume compressed values. These are different operations and measured recipes.

Non-fused INT4 Dequant to an intermediate FP16 buffer, then call standard attention. The temp-buffer write negates the bandwidth savings from reading smaller values. 0.5× (slowdown vs FP16)
Fused INT4 Triton Dequantize K/V inside the attention tile loop — unpacked values live only in registers and never touch global memory. 2.7–4.8× (H100) · 1.6–7.2× (W7900)
Why this stays on the record
The tested unfused path regresses: materializing a temporary full-precision cache adds traffic that consumes the compression benefit. This result motivates fusion; it does not rule out capacity or reuse benefits in other systems that reconstruct data outside the critical decode loop. The fused INT4 Triton kernel proves the mechanism in isolation from FlashInfer's implementation details. Full write-up and kernel source: knlp/docs/fused-quant; appendix of the paper.
Traffic regime

KV capacity and KV bandwidth are different limits.

A cache can fill GPU memory while weight reads still dominate the cost of a decode step. At larger batch or longer context, KV traffic can become a larger share of the work. Grouped-query attention reduces the KV head count and therefore the payload. Model parameter count alone does not determine the crossover.

fKV ≈ B · T · bKV / (W + B · T · bKV + O)

Here T is cached context length, bKV is KV bytes per token across all layers, W is weight traffic per step, and O is other traffic. This is a simplified byte model, not profiler attribution. KV compression has a smaller direct bandwidth opportunity when fKV is small, even if it relieves a capacity limit. Larger B and T also change kernels and execution cost, so confirm the bottleneck with measurements.

This is why fitting only B × T can obscure the deployment question: B controls how many sequences share a decode step, while T controls how much history each sequence brings. Hold T fixed when studying batch saturation, then evaluate the relevant context distribution in serving.

KV ASYM

KV Precision Asymmetry — Keys and Values Are Not the Same

Earlier activation round-trip quantization probes reveal a precision asymmetry: value-only INT4 incurs much less perplexity damage than key-side INT4 in the shown probes. These numerical probes do not demonstrate a native INT4 serving cache or universal INT4 quality safety. Keys exhibit model-dependent precision floors. On Qwen2.5-7B, compressing keys to INT4 (with INT8 values) causes catastrophic collapse — 17,681% PPL increase, 35,000:1 sensitivity asymmetry. On Mistral-7B, the same configuration causes only +1.23% PPL with 0.98 token agreement. The earlier Qwen precision screen placed a sharp degradation between INT7 and INT6 under its quantizer and scaling protocol; that threshold is not a general bit-width rule.

SENSITIVE Qwen2.5-7B

K / V formatΔPPLagreement
KFP16/VFP16—1.00
KFP16/VINT4+0.51%0.94
KINT8/VINT4+2.17%0.93
KINT4/VINT8+17681%0.15
KINT4/VINT4+18305%0.14

TOLERANT Mistral-7B

K / V formatΔPPLagreement
KFP16/VFP16—1.00
KFP16/VINT4+0.44%0.99
KINT8/VINT4+0.21%0.99
KINT4/VINT8+1.23%0.98
KINT4/VINT4+0.87%0.99
Why architectural predictors fail
The tested summary features did not significantly predict key quantization sensitivity in this sample: GQA ratio (ρ=−0.18), RoPE θ (ρ=0.01), attention entropy (r=0.29 at 6 models, down from r=0.89 at 3). Fisher information and covariance-based signals show no significant correlation (Spearman ρ < 0.2, p > 0.3). This negative correlation result does not prove architectural independence. The later failure atlas identifies specific key-bias and partial-RoPE mechanisms. See Spearman ρ explainer.
An earlier calibration screen, with limited scope
Run 5 calibration prompts through INT8 and INT6 key configs (with INT4 values). Compute mean logit error ratio INT6/INT8. Threshold τ = 3.0 classified 13/13 evaluable models correctly in the reported screen. Qwen family: ratio 5.07–5.40 (flag as sensitive). Other tested models: ratio < 2.2. This is evidence for a screening heuristic, not a safety guarantee for unseen models, contexts, formats, or workloads.
Scale attenuation. Qwen key sensitivity attenuates with model scale: catastrophic at 7B (>10% PPL), +49% at 32B, +1.55% at 72B — below practical threshold. Some earlier larger-model INT4 results used weight-quantization proxies and do not establish cache-quantization safety. The direct 72B FP8 activation experiment below is a separate test.
Asymmetric FP16-K / FP8-V rescues fragile-key models without calibration
On Qwen2.5-7B the same fragility observed under INT4 keys reappears under symmetric FP8: WikiText-2 perplexity jumps from 5.2 (FP16) to 774–1046 at 2–16K context under uncalibrated symmetric FP8. Keeping K at FP16 and only quantizing V to FP8 recovers ppl to 5.23–6.07 across the same context range, GSM8K accuracy from 0.0% back to 84.5%, and MMLU from 48.9% back to 76.2% — the screen resolves no difference from the FP16 baseline. No calibration or per-model tuning was required in this experiment. The separate model-level qualification policy determines deployment. See the headline asymmetric section for throughput data and the hardware notes for the separate quality, representation, and hardware execution constraints.
The tested static per-tensor FP8 calibration fails on Qwen2.5-7B
A common reviewer concern is that the symmetric-FP8 baseline is a strawman vs the standard production calibration path. We tested directly with llm-compressor (the KV-quantization calibration tool used in this experiment), 512-sample WikiText-2 calibration set at T=2048, per-tensor static FP8 e4m3 scales loaded by vLLM. On Qwen2.5-7B the result is worse than uncalibrated symmetric FP8: WikiText-2 perplexity jumps to 9.6×105 with GSM8K accuracy at 0%, vs uncalibrated FP8's PPL ≈ 157 / GSM8K 0.5%. The mechanism is the dual of dynamic-calibration failure: a per-tensor scale wide enough to absorb Qwen's K outliers leaves typical channel values rounded to a coarser grid than the unconditional default, destroying bulk precision in exchange for retaining a handful of outlier samples. Asymmetric K16/V8 under the identical protocol yields PPL 7.50 matching FP16 to four decimals and GSM8K 84.5% — an 84.5-point absolute GSM8K improvement over static-calibrated FP8 (84.5% vs 0%). This rejects that per-tensor static calibration protocol. It does not rule out different scale granularities or bias-aware representations; the later pre-bias work repairs retrieval but fails its serving-performance gate.
Hybrid linear + full-attention models — Qwen3.6-27B (Qwen3.5 family)
Qwen3.6-27B uses Qwen3_5ForConditionalGeneration — a hybrid architecture where every fourth layer carries a true KV cache (the rest are linear attention). At T=2048: FP16 PPL 7.358, FP8-sym PPL 7.347 (drops 10.5 absolute points on GSM8K, 35.5% → 25.0%, n=200 8-shot), asymmetric K16/V8 PPL 7.358 with GSM8K 35.5% — sample-by-sample identical to FP16. Asymmetric is also fastest at evaluation time (24 s for the WikiText-2 sweep vs 79 s FP16, 42 s FP8-sym) due to smaller V cache enabling more concurrency. The measured symmetric path remains slower across batch and context: at T=16,384, symmetric FP8 reaches 0.78×–0.80× of FP16 throughput across B ∈ [4, 32]. Interpret these results with the completed kernel and serving decision rather than treating reduced bytes as a universal speed prediction. These results cover the tested hybrid model and protocol. Compressing the attention-layer KV payload does not reduce every layer's recurrent state by the same ratio.
Large-model FP8 activation quant at 72B (closes proxy gap)
Earlier 27B–72B numbers used hook-based weight-quantization as a proxy for activation sensitivity. We now have a direct activation-quant measurement: Qwen2.5-72B-Instruct (FP8-dynamic weights, ~72 GB on H200) on WikiText-103 at T=2048 over 262K tokens. FP16-KV PPL 4.337, FP8-sym PPL 4.355 (+0.41%), asymmetric K16/V8 PPL 4.337 — NLL identical to FP16 to six decimals. The reported aggregate NLL matches at this precision in this 72B activation test. This does not establish identical token distributions or long-context quality, and it does not validate every earlier weight-quantization proxy.
Consistent with KIVI (Liu et al. 2024) which observes channel-wise outlier structure differences between keys and values. Our finding extends from distribution asymmetry to minimum viable bit-width asymmetry. KIVI: arXiv:2402.02750
Hardware scope

Carry the method across GPUs. Refit the curve.

The wider study includes W7900, A100, H100, B200, H200, and MI300X experiments, with different subsets used for memory-traffic characterization, long-context capacity, and dtype comparisons. They are not one matched end-to-end serving matrix. The H100 serving tables and standalone attention microbenchmarks measure different scopes; their token rates should not share an unlabeled leaderboard.

Use sustained bandwidth and measured latency at the target shape. Peak HBM bandwidth alone does not determine decode speed. Kernel dispatch, cache residency, launch overhead, parallelism, and the memory traffic generated by the model all matter. A device's tested context limit is also specific to the model and configuration, rather than a universal GPU limit.

Memory-traffic characterization and measurement scope · Kernel families and hardware constraints.
SPECULATION

Speculation × Quantization — Sub- and Super-Multiplicative Composition

Speculative decoding does not relax bandwidth constraints. The verification step remains a full attention pass over the entire KV cache. Composition with KV quantization is: sub-multiplicative (ρ ≈ 0.62) for aggressively grouped models (Qwen, 4 KV heads) where KV is a small fraction of total bandwidth, and super-multiplicative (ρ up to 1.95) at long context for models with more KV heads (Llama, 8 KV heads) where KV becomes the bottleneck.

ρ = Scombined / (Squant × Sspec)    ρ < 1.0 sub-mult · ρ = 1.0 mult · ρ > 1.0 super-mult
The tested n-gram setup loses benefit at higher batch
At B=1, acceptance ≥ 0.81 through 8K context. At B=16, acceptance is already below 0.65 at 2K context for two of three models. These observations apply to this n-gram workload and scheduler. Batch alone does not determine draft acceptance, and other draft methods need their own measurements.
Tree verification paradox
Tree speculation helps at low acceptance (+21% at α=0.3) but hurts at high acceptance (−17% at α=0.9). This comparison is specific to the tested draft and verification setup; it is not a universal batch threshold for choosing tree or linear speculation.
Offloading and disaggregation

Which bytes cross the link, and how often?

An attention kernel rereading resident KV from GPU memory is a different workload from restoring a cached prefix from NVMe. A third workload transfers prefilled KV to a decode worker. Each has its own transfer frequency, deadline, and concurrency. A per-step bandwidth bound cannot be applied to all three as an SSD requirement.

WorkloadWhen KV movesWhat sets the requirement
Dense decode with streaming overflowNonresident KV needed by attention must be supplied repeatedly.Bytes read per step, active batch, and the decode-step latency budget.
Prefix-cache reload or request resumeMissing KV is restored before the dependent computation; resident reuse avoids repeated storage reads.Miss or resume rate, restored bytes, available overlap, and time-to-first-token or resume latency.
Prefill/decode disaggregationA handoff transfers the required KV between workers.Handoff rate, payload, topology, and transfer deadline. Storage is optional.

A bandwidth example with explicit assumptions

Take the Qwen2.5-7B cache geometry: 28 layers, four KV heads, and head dimension 128. K16/V16 payload is 28 × 4 × 128 × 4 = 56 KiB per cached token. For a hypothetical 131,072-token cache and one sequence, that is 7 GiB. This is byte accounting at an assumed context, not a claim of quality or successful serving at that length.

Calculated payload bandwidth, decimal GB/s; full cache transferred, no metadata or protocol overhead.
RepresentationCache payloadFull reread every 10 msOne reload within 1 s
K16/V167 GiB751.6 GB/s7.52 GB/s
K16/V85.25 GiB563.7 GB/s5.64 GB/s
K8/V8, if qualified3.5 GiB375.8 GB/s3.76 GB/s

The requirement is transferred bytes divided by the available time. Concurrent full rereads multiply the demand; transferring only an offloaded fraction reduces it. A reload service must also sustain its ongoing arrival rate. Pipelining can hide some latency, but it cannot eliminate the bytes or exceed the sustained capacity of the shared path.

What the Saturation Law contributes
The law estimates how much useful decode throughput extra residency could unlock. Offloading can also save prefill recomputation and improve reuse even when the decoder is near its plateau. Transfer overhead and GPU contention can change the curve, so measure the decoder again with offloading enabled.

Prefill/decode disaggregation does not force NVMe offload. Workers can exchange KV through a memory transport. A storage-mediated handoff is an additional design. Direct GPU–NVMe I/O changes the data path and potential host overhead; it does not remove transfer deadlines or create an automatic throughput gain. This page's serving results do not benchmark that path.

GPU pages, transfer groups, and NVMe commands

An attention block's token count, a connector's transfer grouping, and an SSD command's byte size are separate quantities. Packing, coalescing, scatter/gather layout, and command splitting connect them. The same logical KV object can produce different I/O-size and outstanding-I/O distributions.

For storage projections, record cache misses and evictions, bytes per event, transfer deadlines, effective GPU and storage dtypes, packing cost, submitted command sizes, completion latency, and queue depth at the layer being measured. The Saturation Law alone cannot select NVMe maximum transfer size (MDTS), queue depth, SSD count, or predict energy savings.

Model geometry · vLLM disaggregated prefilling · KV I/O projections · I/O tracing.
FUTURE

Future Directions — Reduce KV Access, Not Just Compress It

The paper's closing argument: compression and kernel engineering are the current best path, but future architectures should address the KV bandwidth bottleneck more directly — constraining KV access proportional to available memory bandwidth rather than compressing the entire cache. Two concrete external directions that motivate this:

MXFP8 on Blackwell
Block-scaled FP8 needs a new quality and execution comparison
K16/V8 protects key precision and changes operand preparation. New hardware can change that execution tradeoff, but does not automatically remove model-level key sensitivity. Blackwell adds block-scaled MXFP8 with hardware consumption of per-block scales for narrow Q and K. FlashInfer contains an SM100 block-scaled fused-attention implementation. Validate its paged-cache layout, model shapes, numerical policy, and serving dispatch against plain E4M3 transformation and K16/V8.
Production R&D
What Remains to Ship
Upstream integration: Land the asymmetric allocator, cache writer, decode kernel, serialization, and API path in the serving projects that own them.

Capacity validation: Compare the representation-aware byte model with allocator telemetry, fragmentation, prefix caching, sliding windows, offload, and disaggregated serving.

Runtime evidence: Have engines attest the effective key/value dtype and carry it through benchmark provenance. Extend model, hardware, tensor-parallel, cache-lifecycle, and long-context coverage without reopening the closed pre-bias kernel line.
Current model-level policy
Use symmetric K8/V8 only after the model and workload pass quality qualification. Keep K at 16 bits when that key quantization fails and K16/V8 passes; retain the full-precision reference when neither tier is qualified. A family name or attention_bias=false is insufficient evidence. The bias-aware policy records the closed pre-bias kernel decision: retrieval was repaired, but the tested equal-memory serving comparison returned 0.654× request throughput and 2.056× p95 request latency relative to K16/V8. That implementation is not the serving default. Trained cartridges require their own qualification and matched sharing baseline; ordinary-generation scores do not establish cartridge quality.
Evidence archive

Earlier measurements, with their original scope

The current result is above. This archive preserves the earlier batch-8 gains, eight-model batch-32 matrix, failed attempts, historical quality tables and dated video. Source changes, workloads and quality contracts differ; percentages from these studies cannot be added.

Open earlier results and source history
OCT 2026

October 2026: qualified batch-8 serving gains

The optimized serving stack combines a smaller persistent KV cache with a modest complete-request throughput gain in two tested recipes. On one H100 SXM 80 GB at TP1, with 8,192 input tokens, batch 8 and 128 generated tokens, Qwen3-8B-AWQ improves by 2.28% over matched native FP16 KV. Dense Qwen3-8B executing in FP16 improves by 2.49%. Both pass the frozen NLL, retrieval and full GSM8K preservation tests. These are workload-specific serving results. The completed B32 retest below gives separate model and batch results.

October complete-request throughput: median within-block ratios from seven fresh three-arm process blocks per recipe, separate from pilots. Native means 16-bit K and V.
Checkpoint and actual executionCompact / nativeFrozen quality decision
Qwen3-8B-AWQ: packed INT4 weights, FP16 execution1.022845850× (+2.28%)NLL, retrieval and all 1,319 GSM8K questions pass
Qwen3-8B: dense FP16 weights and execution1.02486566× (+2.49%)NLL, retrieval and all 1,319 GSM8K questions pass
Qwen2.5-7B-Instruct: BF16 weights and execution1.006177719× (+0.62%)GSM gross-loss preservation fails; experimental

Every primary ratio favors compact. Ratios within a process block are correlated. The same weight checkpoint is used across each experiment's cache arms: AWQ weights remain packed. The first two native controls use FP16 KV. Timing pools are fixed at 48 GiB for AWQ and 32 GiB for dense FP16 and BF16. APC and chunked prefill are off. These results compare complete serving recipes, rather than an isolated attention kernel.

Make prefill cheap enough to expose the decode benefit

Earlier writer, scale handling and decode graph work improved decode, but long-query prefill still held back complete requests. The compact route gathers stored K16/V8 pages into one bounded, shared FP16 workspace for native FA3 prefill. The values were already quantized: casting them to FP16 does not restore pristine values. It retains no full per-layer FP16 V shadow. Decode continues to consume compressed persistent values. In the controlled AWQ study, compact is 6.19% faster than the preceding mixed stack and 2.28% above native for complete requests. This spans vLLM allocation, dispatch, writer/scale/graph handling and FlashInfer; percentages from separate campaigns must not be added.

Memory and capacity are separate from the fixed-B8 throughput measurement.
QuantityMeasured or calculated scope
Persistent K16/V8 payload0.75× native at equal token slots: 25% less KV payload, not total GPU memory
Compact temporary storage, Qwen3296.140625 MiB constant GPU workspace, including 256 MiB shared payload; variable metadata and 8 MiB pinned CPU storage. Count aliases once.
Separate automatic-pool AWQ resident endpointsCompact 58 · preceding mixed 59 · native 44 requests. Compact admits 31.82% more than native after workspace costs.

A fixed byte pool can spend saved payload on additional token slots, so total reserved memory need not fall. The automatic-pool endpoints measure resident capacity, not throughput, queueing or cost per token. Do not multiply that capacity ratio by the fixed-B8 speed ratio.

Limits of the preceding compact recipe
Before the mixed-batch routing change, the descriptive AWQ 8K/B32 compact/native ratio was 0.985629×, about 1.44% slower, with mixed prefill/decode fallback. The current route has a separate qualified B32 gain. The BF16 transfer has a slightly higher net GSM score, but its gross-loss upper bound exceeds the frozen limit; it remains experimental. A separate Q8/K8/V8 AWQ recipe fails quality and has no comparative throughput measurement. These are recipe-specific outcomes, not a K-only mechanism proof or a verdict on all symmetric policies.

Completed: the matched batch-32 test. The full findings below retain all eight primary models and four bridge cells, including quality failures, a production failure and inconclusive timing. Two primary recipes qualified for complete paired timing; neither met the predeclared gain rule. The B8 gain does not replace any historical B32 row. Exact old-to-new deltas remain unavailable because the April provenance is incomplete.

Dated source and reproduction

Exact measured dependencies, published October 16. Controls remain reachable through vLLM ancestry.
DependencyPublic branchImmutable measured commit
FlashInfer20261016-asym-k16v8-serving800a550
vLLM compact prefill20261016-k16v8-compact-prefill26cd429
vLLM preceding mixed / native controlsSame dated branch ancestry1a4dcc0 / c21c087

These exact commits extend the July 6 full mixed-cache serving bases, separately from the later pre-bias work. See the separate October reproduction configuration for source/build pins, public input and scale generation, graph capture, scheduling, CPU threads, quality gates, memory accounting and timing boundaries. Historical defconfig defaults remain separate.

October 16 video update

Historical October 16 presentation, recorded before the B32 matrix and the current routing gain · completed B32 findings · stock synthetic narration · text transcript · captions · numerical data.
B32 RETEST

Earlier implementation: October batch-32 matrix

2 of eight primary models qualified for the paired timing study; 0 met the predeclared gain rule. Every planned model remains visible below. Tables scroll horizontally on small screens. Failed production or quality gates have no qualified throughput ratio.

One H100 SXM 80GB, tensor parallel size 1, FP16 weights and activations, 22 CPU threads, 32 concurrent requests with 1,024 frozen input token IDs and 128 forced outputs each. Native stores keys and values in FP16; asymmetric stores FP16 keys and FP8 E4M3 values. Prefix caching and chunked prefill are off. Primary native KV uses FlashAttention 3; asymmetric KV uses FlashInfer with its supported compact or mixed route. These are complete serving-recipe comparisons, including the backend difference.

Primary results. Ratios are asymmetric/native complete-request throughput. Brackets show the descriptive, unadjusted 95% paired-block bootstrap interval for the median.
ModelAsymmetric routeConfigured KV poolOutcomeMedian ratio [95% CI]Holm p, family 24
Qwen/Qwen3-8Bcompact32 GiBNo confirmed gain1.00264× [1.00090, 1.00488]0.257812
Qwen/Qwen2.5-7B-Instructmixed32 GiBFails NLL marginN/A1
Qwen/Qwen2.5-0.5B-Instructmixed8 GiBD64 production decode/reference check failedN/A1
Qwen/Qwen2.5-1.5B-Instructmixed8 GiBFails GSM preservationN/A1
mistralai/Mistral-7B-Instruct-v0.3compact32 GiBFails native GSM floor, GSM preservationN/A1
meta-llama/Llama-3.1-8B-Instructcompact32 GiBFails GSM preservationN/A1
microsoft/Phi-4mixed24 GiBNo confirmed gain0.97355× [0.97246, 0.97553]1
deepseek-ai/DeepSeek-R1-Distill-Qwen-7Bmixed32 GiBFails native retrieval floor, retrieval preservationN/A1
Strict quality panel. NLL bounds resample independent documents. Retrieval and GSM show native → asymmetric scores and exact one-sided 95% gross-loss upper bounds.
ModelMean ΔNLL [upper]RetrievalGSM8K
Qwen/Qwen3-8B+0.000263 [upper +0.001287]100/100 → 100/100; 0 losses, 0 gains; loss upper 2.951%1175/1319 → 1178/1319; 11 losses, 14 gains; loss upper 1.377%
Qwen/Qwen2.5-7B-Instruct+0.055686 [upper +0.070457]100/100 → 100/100; 0 losses, 0 gains; loss upper 2.951%Not run
Qwen/Qwen2.5-0.5B-InstructNot runNot runNot run
Qwen/Qwen2.5-1.5B-Instruct+0.000951 [upper +0.001974]88/100 → 88/100; 1 losses, 1 gains; loss upper 4.656%720/1319 → 685/1319; 129 losses, 94 gains; loss upper 11.231%
mistralai/Mistral-7B-Instruct-v0.3-0.000051 [upper +0.000399]73/100 → 73/100; 0 losses, 0 gains; loss upper 2.951%314/1319 → 306/1319; 42 losses, 34 gains; loss upper 4.099%
meta-llama/Llama-3.1-8B-Instruct-0.001505 [upper -0.000306]100/100 → 100/100; 0 losses, 0 gains; loss upper 2.951%874/1319 → 870/1319; 20 losses, 16 gains; loss upper 2.196%
microsoft/Phi-4-0.000690 [upper -0.000238]100/100 → 100/100; 0 losses, 0 gains; loss upper 2.951%1252/1319 → 1253/1319; 3 losses, 4 gains; loss upper 0.587%
deepseek-ai/DeepSeek-R1-Distill-Qwen-7B-0.008660 [upper -0.003593]49/100 → 54/100; 7 losses, 12 gains; loss upper 12.746%Not run

Quality precedes timing: 32 independent NLL documents with 128 cached-decode targets each, 100 exact retrieval cases, and all 1,319 questions in the frozen strict GSM8K chat adaptation when earlier gates pass. NLL upper Δ must be at most ln(1.01), retrieval gross-loss upper at most 5%, and GSM gross-loss upper at most 2%. Errors and truncated answers are wrong. Native quality floors are fixed per model and are included in the data file. A failed native floor limits the conclusion to this scored recipe.

Eligible primary cells use ten fresh process pairs in a frozen, counterbalanced order. A gain requires all gates, median ratio at least 1.01, and a positive one-sided sign test after Holm correction over all 24 planned comparisons. Missing historical comparisons retain p=1. Ties are non-wins. No old-to-new speedup delta is inferred from the April table.

All retained timing explicitly enables request statistics in both arms. Phi-4 and AWQ use knlp a6016292; corrected dense Qwen3 cells use 720bc026. The earlier recorder received no request metrics and produced no valid paired timing. Those attempts are excluded. Phi-4 retains its complete quality panel from c4bb3827: scoring, model math, inputs, scales and routes are unchanged, and both corrected arms pass fresh production profiles. No valid timing is pooled across source versions.

An input audit invalidated the first dense Qwen3 attempts: preparation had copied a native-only unit-scale placeholder, so those quality decisions and timings are excluded. The affected dense cells were rerun with the exact already-accepted calibrated scales, verified against the prior scored protocols. No scales were tuned against the new answers. The corrected source rejects placeholders before serving. All unaffected primary and AWQ measurements are retained.

Separate 8K bridge

Both bridge native and asymmetric recipes use FlashInfer. Dense weights and activations are FP16; AWQ weights stay packed INT4 with FP16 activations. Each eligible cell uses three fresh process pairs at 8,192 input tokens and 128 forced outputs. These ratios are descriptive and do not enter the primary family. The compact variant retains guarded fallback for mixed prefill/decode calls; the profile counters are included in the data file. Accepted quality is reused only for the unchanged dense B8 and AWQ recipes, after input, scale, source and actual B32 route checks; production profiles are fresh. Dense B32 receives fresh quality checks at its frozen 40 GiB pool, including full GSM when earlier gates pass.

Descriptive October bridge, with all four planned cells retained.
Model / weightsBatchConfigured KV poolQuality evidenceMedian ratio [95% CI]Outcome
Qwen/Qwen3-8B832 GiBReused accepted quality; fresh production1.01934× [1.01914, 1.02073]Descriptive bridge
Qwen/Qwen3-8B3240 GiBFresh quality; fresh production0.98782× [0.98574, 0.99132]Descriptive bridge
Qwen/Qwen3-8B-AWQ848 GiBReused accepted quality; fresh production1.02220× [1.02110, 1.02350]Descriptive bridge
Qwen/Qwen3-8B-AWQ3248 GiBReused accepted quality; fresh production0.98833× [0.98727, 0.98844]Descriptive bridge

The timer excludes process startup, build and one fixed warmup, then synchronizes CUDA at both complete-request boundaries. It includes prefill, scheduling and decode. Completion-linked records separately verify actual full-batch decode; those host completion intervals are not isolated GPU kernel time. Equal byte pools can allocate different token-slot counts. The 25% K16/V8 payload saving at equal slots is not a 25% reduction in total GPU memory, and capacity ratios are not multiplied by throughput ratios.

Qwen/Qwen3-8B · 1024 input tokens · B32: all retained paired ratios

0.9935141×, 1.0046516×, 1.0002696×, 1.0014335×, 1.0058214×, 1.0016846×, 1.0039351×, 1.0036023×, 1.0068204×, 1.0015314×.

Median arm rates: native 2146.93 output tokens/s; asymmetric 2154.13 output tokens/s. The primary comparison uses within-pair ratios, rather than a ratio of these medians.

microsoft/Phi-4 · 1024 input tokens · B32: all retained paired ratios

0.9736099×, 0.9712515×, 0.9746203×, 0.9733828×, 0.9734810×, 0.9757707×, 0.9715308×, 0.9759696×, 0.9730844×, 0.9755253×.

Median arm rates: native 1276.18 output tokens/s; asymmetric 1243.18 output tokens/s. The primary comparison uses within-pair ratios, rather than a ratio of these medians.

Qwen/Qwen3-8B · 8192 input tokens · B8: all retained paired ratios

1.0191417×, 1.0193403×, 1.0207322×.

Median arm rates: native 324.31 output tokens/s; asymmetric 330.59 output tokens/s. The primary comparison uses within-pair ratios, rather than a ratio of these medians.

Qwen/Qwen3-8B · 8192 input tokens · B32: all retained paired ratios

0.9913167×, 0.9878161×, 0.9857369×.

Median arm rates: native 412.94 output tokens/s; asymmetric 408.61 output tokens/s. The primary comparison uses within-pair ratios, rather than a ratio of these medians.

Qwen/Qwen3-8B-AWQ · 8192 input tokens · B8: all retained paired ratios

1.0221985×, 1.0234999×, 1.0211044×.

Median arm rates: native 338.49 output tokens/s; asymmetric 346.40 output tokens/s. The primary comparison uses within-pair ratios, rather than a ratio of these medians.

Qwen/Qwen3-8B-AWQ · 8192 input tokens · B32: all retained paired ratios

0.9884363×, 0.9883347×, 0.9872666×.

Median arm rates: native 406.56 output tokens/s; asymmetric 401.78 output tokens/s. The primary comparison uses within-pair ratios, rather than a ratio of these medians.

Complete derived data, all retained ratios, model pins and frozen floors · Original quality source · Phi/AWQ timing source · Dense calibration guard and runtime · October reproduction configuration.

R

October 16 repair checks — neither candidate qualified

These October 16 attempts precede the completed routing and D64 qualifications. The mixed-prefill metadata candidate stopped in its initial all-prefill admission, before any mixed decode/prefill batch. Across 34 completed layers, full output and LSE matched the existing mixed-cache fallback bit for bit. One of 417,792 fixed sampled outputs exceeded the independent stored-value reference bound. CPU replay reproduces the discrepancy and an FP16 probability-rounding diagnostic. This does not establish a routing defect. Fresh quality and comparative timing were not reached.

The separate D64 layout candidate passed CPU address-coverage checks and compiled four CUDA layout instantiations. Its numerical JIT check stopped before output because the launcher omitted the environment's ninja directory from PATH. At that point, the original D64 failure and candidate repair had not been numerically tested in that follow-up; the earlier Qwen0.5 failure row retains its scope.

Both pods are deleted after verified exports and CPU replay. Total use was 0.33449 GPU-hours, with a provisional $1.35 estimate including disk allowance. Neither check produced a new speed result or qualified dependency release. Exact scope, sources and failed gate.

Serving evidence

Historical Qwen2.5-7B serving results — source identity unresolved

The table preserves the previously reported Qwen2.5-7B serving measurements. Their exact full-stack source identity remains unresolved. The July 2 vLLM 758786e2 and FlashInfer ff0c51cf links identify decode-only lineage; that vLLM revision rejects asymmetric prefill and cannot substantiate this prompt-to-generation recipe. The historical report compares controlled FP16 / FP8-sym / K16/V8 measurements on the same H100 path: K16/V8 matches FP16 perplexity to reported precision at both 2K and 8K context, and the 200-problem GSM8K screen did not resolve a difference between K16/V8 (90.0%) and FP16 (90.5%) — a screen of this size can detect collapse (symmetric FP8: 2.0%) but cannot establish equivalence.

Config PPL@2K PPL@8K GSM8K (n=200) Smoke tok/s
FP16 baseline 6.997 5.243 90.5% 102.6
FP8 symmetric 214.3 1058.0 2.0% 97.9
Asymmetric K16/V8 6.997 5.243 90.0% 105.8

Symmetric FP8 PPL gets 5× worse from 2K to 8K (214→1058) — the K-fragility phenomenon strengthens at longer context. Asymmetric K16/V8 matches FP16 perplexity to the reported precision at both context lengths. K16 denotes native 16-bit keys. Historical FP16 labels are retained for their original comparisons; source gaps prevent treating every table as an identical baseline. These are historical reported serving results with unresolved source provenance; the earlier HuggingFace DynamicCache simulation in the precision-asymmetry section is a separate numerical cross-check.

Tensor parallelism was tested at TP = 1, 2, and 4 on a 4×H100 pod. The reported 48-sequence test found bit-identical prefill NLL and 100% autoregressive token agreement for K16/V8 versus the reference at each tested degree. At TP = 4, the four KV heads shard to one head per rank. These are finite-sample correctness observations, not proof of zero quality cost on all workloads. The source identity for these historical checks also remains unresolved. Prefill agreement alone does not qualify reads from a quantized cache.

Historical reported implementation checks; exact builds unresolved
Gate Result
FlashInfer decode, BF16-K / FP8-V rel err 0.0254 vs BF16 ref
FlashInfer prefill, BF16-K / FP8-V rel err 0.0255 vs BF16 ref
vLLM cache writer K bit-exact, V within FP8 noise
K/V aliasing check disjoint storage
Required backend FlashInfer via attention_config

For the current qualified serving implementation, use the current routing reproduction method and its immutable source pins. That Qwen3 recipe is separate from the historical Qwen2.5 measurements above and does not recover their original runtime. The July 2 decode-only links are not an end-to-end reproduction command.

Earlier kernel and quality evidence

K16/V8 — A Qualified Option for Fragile Keys

Storing keys at native 16-bit precision and only quantizing values to FP8 removes FP8 K operand preparation before each tile's QK operation. Account for FP8 V preparation before PV and measure how much the kernel overlaps it. On the standalone CUDA-core FlashInfer decode kernel on the headline Qwen2.5-7B lane, asymmetric K16/V8 is close to the FP16 reference in this measurement and leads the measured symmetric FP8 path; on the SM90 Tensor Core kernel family the sign reverses — symmetric K8/V8 runs 16–19% faster than K16/V8 at bandwidth-bound shapes (see the linked hardware notes). Across the eight-model vLLM batch-32 comparison below, asymmetric is up to 4.2% faster than symmetric FP8 on the models where it leads. K16/V8 provides 1.33× the KV payload capacity of K16/V16, but only two-thirds of K8/V8 capacity. This implementation uses modified vLLM/FlashInfer kernels without changing model weights or retraining. Quality qualification remains necessary.

Conventional equally sized K/V tensors; raw payload ratios only.
LayoutBytes per K/V pairKV capacity vs K16/V16Qualification
K16/V1641.00×Full-precision reference
K16/V831.33×Retains key precision; qualify value compression
K8/V822.00×Qualify both keys and values for the workload
Mechanism — why bytes alone do not predict throughput
Hopper has no native BF16-query by plain-FP8-key QK instruction with a K scale operand. Prepare each active K tile on chip before QK and advance online softmax tile by tile. Compare the unhidden preparation and occupancy cost against the extra K16 HBM byte. Account for V preparation before PV. CDNA 3 provides FP8-by-FP8 MFMA with no scale operands; place the scale in software around MFMA. Use the completed bias-aware KV findings and the hardware notes.
Quality rescue on fragile-key models
NIAH multikey-3 retrieval at 32K tokens on H100 via vLLM + FlashInfer, across three models:
Qwen2.5-7B (fragile)   FP16: 0.84  ·  FP8-sym: 0.00  ·  Asym K16/V8: 0.89
Qwen3-8B (tolerant)   FP16: 0.95  ·  FP8-sym: 0.92  ·  Asym K16/V8: 0.95
Llama-3.1-8B (tolerant)   FP16: 1.00  ·  FP8-sym: 1.00  ·  Asym K16/V8: 1.00
Qwen2.5-7B scores zero under symmetric FP8 in this retrieval test. The tested K16/V8 retrieval score is 0.89 versus 0.84 for the Qwen2.5-7B reference. Without an uncertainty estimate, this difference does not establish superiority. The two other models show no collapse in this test. When one qualified KV-cache scheme must cover these tested robust- and fragile-key models, asymmetric K16/V8 is the preferred option among the evaluated fused FP8 paths: it is the only one here that holds quality on the fragile models while matching the tolerant ones. The later diagnosis helps explain these observations: symmetric FP8-K collapse tracks K-projection bias magnitude (with a distinct partial-RoPE failure mode), so on a known-robust model symmetric FP8-K is quality-admissible and, being denser, can be the better choice. See the FP8 KV-cache failure atlas for the root cause and the per-model fragility map. A fuller comparison of K16/V8 against the broader KV-quantization field — symmetric FP8, pre-bias, sub-8-bit values, and other approaches across quality, capacity, kernel cost, and served goodput — is now available in the KV-cache compression frontier.
Code: github.com/mcgrof/flashinfer · serving branch 20260702-asym-k16v8-decode-upstream (fork commit ff0c51cf; curated decode-only K16/FP8 dtype-split series). vLLM-side plumbing on 20260702-k16fp8 (fork commit 758786e2). The asym-prefill-refactor-stage dev tip carries a five-commit CUDA template refactor giving independent K and V dtypes in prefill and decode; its mixed-dtype prefill kernels compile and pass fp32-reference audits, but it is not part of the validated serving pair. Integrates with vLLM via LLM(kv_cache_dtype=("auto", "fp8_e4m3"), attention_config={"backend": "FLASHINFER"}).
THROUGHPUT

Historical April 2026 batch-32 throughput

These eight published values match an April 16, 2026 results file, rather than an established July run. The retained logs show eager FP16 execution, APC and chunked prefill enabled, a 16,384-token scheduler budget, native FLASH_ATTN/FA3 and asymmetric FLASHINFER. Exact source, checkpoint/tokenizer revisions, prompt IDs, successful Qwen3/Phi logs and timer boundaries are not fully recovered. The reported sequence-length field does not attest actual prompt length. Treat these as historical serving measurements, not a matched comparison with the October stack.

The Qwen2.5-7B asymmetric/native ratio is 1.377×. Retained native logs show about 54.53 GiB of KV capacity and 1,020,992 token slots, above this B32 workload. They do not establish the earlier explanation that native capacity pressure caused that ratio. The remaining asymmetric/native ratios range from 0.871× to 1.029×. Cause requires a matched workload and profiling.

Published H100 batch-32 serving throughput. Ratios use the 16-bit baseline in the same row.
Model16-bit KV (tok/s)K8/V8 (tok/s)K16/V8 (tok/s)K16/V8 / baseline
Qwen2.5-0.5B3,312.33,098.92,884.40.871×
Qwen2.5-1.5B2,912.92,741.32,857.40.981×
Qwen2.5-7B2,065.72,763.02,845.01.377×
Mistral-7B2,861.82,915.92,945.41.029×
Llama-3.1-8B2,084.92,085.52,105.21.010×
Qwen3-8B2,467.32,282.82,315.50.938×
Phi-4-14B1,027.61,041.51,018.40.991×
DS-R1-Qwen-7B2,979.52,896.42,866.40.962×
Preserve the observed ordering; leave its cause open
Asymmetric leads symmetric FP8 in five of eight historical rows and trails it in three. This fixed-batch table does not isolate a kernel mechanism or prove physical byte savings. Quality is a separate model/workload condition; retrieval evidence elsewhere on this page does not qualify every cell in this table. An equal-memory capacity test answers a different question.
Historical evidence identifier: 24d34902e272a293409f0b9f90d362344252f342, h100_throughput_sweep.json. The printed later branch pins are not attested as the source of these April rows. The completed October matrix retains this table and reports its own workload and provenance.
REP

Reproduce the current results

Use the exact serving pair below for both qualified 8K/batch-32 results. Branches are public; immutable commits identify the measured code. The AWQ method and dense FP16 method record their respective model, scale and harness pins, KV pools, controls and quality gates.

Current routing recipe. Keep the separate D64 repair out of this speed comparison.
ComponentPublic branchMeasured commit
vLLM: mixed-batch fresh prefill2026-10-17-feature-mixed-fresh-prefill276305b2
FlashInfer: measured serving kernels20261016-asym-k16v8-serving800a550f
knlp: AWQ quality and timing harnessImmutable sourcef91b3c3f
knlp: dense FP16 quality and timing harnessImmutable sourcea3996c8f

The separate D64 repair uses FlashInfer 2026-10-17-fix-mixed-d64-paged-layout at 72b18b4f, with vLLM 26cd429b for its earlier synthetic suite. The repaired full-model production check uses vLLM 1a4dcc01; retrieval still blocks model qualification. Combining that repair with the new routing branch is an unqualified source pair.

Earlier reproduction profiles and source lineage

Use the October configuration for the preceding compact serving results. The decode-only lineage branches below are retained for historical reference. They do not establish the full-stack source of the legacy quality tables or the April B32 table. On October 16, the July 2 FlashInfer branch resolved to ff0c51cf676bf4461695c34af564461f1e30c0a6 and the July 2 vLLM branch to 758786e23230a4a1a9b2b6046fb516943751b3ae.

The formerly printed pins instead belong to the later pre-bias branches: FlashInfer 2026-08-22-feature-prebias-k8-cache (6dfdc833) and vLLM 2026-08-22-feature-prebias-k8-vllm (8a171410). These are separate historical pointers, not October measured dependencies.

Repo Branch What it contains
mcgrof/vllm 20260702-k16fp8 (fork commit 758786e2) Decode-only lineage: K16/V8 tuple cache and asymmetric decode; asymmetric prefill raises NotImplementedError. This revision does not reproduce the legacy serving table
mcgrof/flashinfer 20260702-asym-k16v8-decode-upstream (fork commit ff0c51cf) Dated fork implementation: curated decode-only K16/FP8 dtype-split series, rebased on upstream main, GPU-gated for the upstream PR ladder
mcgrof/LMCache asymmetric-kv-codec K16/V8 codec, split-tier placement, serde, 74 CPU unit tests
Planning and result provenance: aiconfigurator K16/V8 byte accounting v2 prices conventional K16/V8 caches without claiming unmeasured latency and rejects compressed-latent layouts. InferenceX KV dtype provenance v2 records requested key/value dtypes without retaining raw server command lines and requires runtime attestation before calling a dtype effective.

The knlp defconfig system selects and runs the reproduction workflow. Review its configuration before building: 

git clone https://github.com/mcgrof/knlp.git && cd knlp
make defconfig-decode-upstream   # Legacy decode-only profile; inspect supported stages
make

defconfig-decode-upstream selects the legacy decode-only branches above; it is not a reproduction of the reported full-stack quality results. Use the current routing method above for the qualified batch-32 serving recipe. Branch names can move; record the resolved commit IDs and actual supported stages. The reproduction profiles are: defconfig-decode (core full-stack quality battery + standalone FlashInfer gates + LMCache codec checks), defconfig-decode-sat (saturation model + Hill fit), and defconfig-decode-full (everything possible from the paper, with structured skip reports when cross-GPU hardware is missing). These profiles select git refs, which may be branches rather than immutable commits, clone the companion forks into the parent directory, build the modified serving stack, run the configured stages, and write machine-readable artifacts to results/decode/<run_id>/. Local JSONL telemetry is canonical; W&B and trackerio are optional mirrors. See docs/reproduce/paper-memory-decode.md for the workflow and hardware requirements. These are fork-specific research recipes, not instructions for an unmodified upstream installation. Fresh prefill, cached-prefix prefill, and decode require separate correctness coverage; full-precision fresh prefill alone does not validate mixed-dtype prefix reads.