knlp / bias-aware KV quantization

Completed research · negative serving result

Bias-aware FP8 repairs retrieval.
Its serving cost is too high.

Can we fix the key-quantization failure, keep both keys and values in FP8, and make the correction fast enough to serve? The tested pre-bias representation repairs retrieval. Its optimized kernel delivers 35% fewer completed requests per second than asymmetric K16/V8 in the equal-memory test, with 2.06× p95 request latency.

This page covers that closed pre-bias experiment. Current asymmetric K16/V8 results, supported models, and installation are on the asymmetric quantization guide.

The representation repairs the tested failure

Ordinary FP8 key storage destroys long-context retrieval on the tested Qwen2.5-7B model. Pre-bias stores bias-removed keys in FP8 and restores the position-dependent bias contribution inside attention. Values also remain in FP8. The correction recovers the information that ordinary key quantization loses.

Qwen2.5-7B: needle-retrieval score at each tested input length. Higher is better.
KV representation8,192 tokens16,384 tokens30,720 tokens
Ordinary FP8 keys / FP8 values0.000.000.00
Pre-bias FP8 keys / FP8 values1.001.001.00
16-bit keys / FP8 values1.001.001.00
16-bit keys / 16-bit values1.001.001.00

The optimized correction also passes all eight clean mutation-test cases at the model's attention geometry; injected defects that affect that kernel are detected. Pre-bias provides 1.50× the measured KV token capacity of K16/V8.

These results establish retrieval rescue and tested kernel correctness. They do not establish general quality equivalence. The 200-problem GSM8K math screen reports 45.5% for pre-bias and 49.5% for the 16-bit reference; only aggregate scores were retained, so paired outcome analysis cannot be recovered.

More capacity does not produce more served work

The final test gives each method the same memory budget, then compares its capacity point: 32 concurrent requests for K16/V8 and 48 for pre-bias. It uses Qwen2.5-7B on one H100, identical seeded prompts, 30,720 input tokens and 128 fixed output tokens. Both arms use FlashInfer's generic kernel family.

Equal-memory serving result. Ratios compare pre-bias with K16/V8; throughput uses the ratio of the two block means.
MetricK16/V8Pre-bias FP8Pass rule
Concurrent requests3248Use each capacity point
Completed requests / second0.6561670.428922—
Throughput ratio1.000×0.654×At least 1.20×
p95 request-latency ratio1.000×2.056×At most 1.25×
The optimized pre-bias kernel fails both serving conditions.Pre-bias remains near 0.43 requests per second at concurrencies 16, 24, 32, and 48. The sweep points to a prefill bottleneck: extra cache capacity adds waiting without increasing completed work.

Separate kernel measurements also show a cost: pre-bias takes 1.687× the 16-bit-key kernel time at 4,096 tokens and 1.107× at 32,768 tokens. Five fresh processes confirm the pattern at the served geometry. Smaller storage alone does not guarantee faster attention.

The serving evidence contains two agreeing interleaved blocks, rather than the five planned repetitions. Both throughput and latency miss their bounds by wide margins, so the review accepts the negative engineering decision. This is not a five-repetition performance certification or a claim about every future bias-aware implementation.

Evidence and scope

The served geometry is 28 query heads, 4 key/value heads, and rotary base 1,000,000. The measured source versions are vLLM 8a1714108 and FlashInfer 6dfdc833. These are the experimental pre-bias versions, not the current asymmetric installation recipe.

Serving blocks behind the decision
Each block compares K16/V8 at concurrency 32 with pre-bias at concurrency 48.
BlockK16/V8 requests/sPre-bias requests/sThroughput ratiop95 latency ratio
10.6573970.4288390.6523×2.061×
20.6549360.4290040.6550×2.051×

The conclusion applies to this representation, kernel family, model, hardware, and workload. The quality repair and measured capacity remain valid results. Further tuning of this kernel line is closed; a materially different representation, kernel family, or hardware target would require a new study.

Use the current asymmetric guide

K16/V8 keeps the keys at 16-bit precision and compresses values to FP8. Its current performance and quality claims are qualified per model and workload.