Completed research · negative serving result
Bias-aware FP8 repairs retrieval.
Its serving cost is too high.
Can we fix the key-quantization failure, keep both keys and values in FP8, and make the correction fast enough to serve? The tested pre-bias representation repairs retrieval. Its optimized kernel delivers 35% fewer completed requests per second than asymmetric K16/V8 in the equal-memory test, with 2.06× p95 request latency.
This page covers that closed pre-bias experiment. Current asymmetric K16/V8 results, supported models, and installation are on the asymmetric quantization guide.
The representation repairs the tested failure
Ordinary FP8 key storage destroys long-context retrieval on the tested Qwen2.5-7B model. Pre-bias stores bias-removed keys in FP8 and restores the position-dependent bias contribution inside attention. Values also remain in FP8. The correction recovers the information that ordinary key quantization loses.
| KV representation | 8,192 tokens | 16,384 tokens | 30,720 tokens |
|---|---|---|---|
| Ordinary FP8 keys / FP8 values | 0.00 | 0.00 | 0.00 |
| Pre-bias FP8 keys / FP8 values | 1.00 | 1.00 | 1.00 |
| 16-bit keys / FP8 values | 1.00 | 1.00 | 1.00 |
| 16-bit keys / 16-bit values | 1.00 | 1.00 | 1.00 |
The optimized correction also passes all eight clean mutation-test cases at the model's attention geometry; injected defects that affect that kernel are detected. Pre-bias provides 1.50× the measured KV token capacity of K16/V8.
These results establish retrieval rescue and tested kernel correctness. They do not establish general quality equivalence. The 200-problem GSM8K math screen reports 45.5% for pre-bias and 49.5% for the 16-bit reference; only aggregate scores were retained, so paired outcome analysis cannot be recovered.
More capacity does not produce more served work
The final test gives each method the same memory budget, then compares its capacity point: 32 concurrent requests for K16/V8 and 48 for pre-bias. It uses Qwen2.5-7B on one H100, identical seeded prompts, 30,720 input tokens and 128 fixed output tokens. Both arms use FlashInfer's generic kernel family.
| Metric | K16/V8 | Pre-bias FP8 | Pass rule |
|---|---|---|---|
| Concurrent requests | 32 | 48 | Use each capacity point |
| Completed requests / second | 0.656167 | 0.428922 | — |
| Throughput ratio | 1.000× | 0.654× | At least 1.20× |
| p95 request-latency ratio | 1.000× | 2.056× | At most 1.25× |
Separate kernel measurements also show a cost: pre-bias takes 1.687× the 16-bit-key kernel time at 4,096 tokens and 1.107× at 32,768 tokens. Five fresh processes confirm the pattern at the served geometry. Smaller storage alone does not guarantee faster attention.
The serving evidence contains two agreeing interleaved blocks, rather than the five planned repetitions. Both throughput and latency miss their bounds by wide margins, so the review accepts the negative engineering decision. This is not a five-repetition performance certification or a claim about every future bias-aware implementation.
Evidence and scope
The served geometry is 28 query heads, 4 key/value heads, and rotary base 1,000,000. The measured source versions are vLLM 8a1714108 and FlashInfer 6dfdc833. These are the experimental pre-bias versions, not the current asymmetric installation recipe.
Serving blocks behind the decision
| Block | K16/V8 requests/s | Pre-bias requests/s | Throughput ratio | p95 latency ratio |
|---|---|---|---|---|
| 1 | 0.657397 | 0.428839 | 0.6523× | 2.061× |
| 2 | 0.654936 | 0.429004 | 0.6550× | 2.051× |
The conclusion applies to this representation, kernel family, model, hardware, and workload. The quality repair and measured capacity remain valid results. Further tuning of this kernel line is closed; a materially different representation, kernel family, or hardware target would require a new study.
Use the current asymmetric guide
K16/V8 keeps the keys at 16-bit precision and compresses values to FP8. Its current performance and quality claims are qualified per model and workload.