The selected kernel family changes the FP8 ordering.
The diagnostic CUDA-core path makes every tested FP8 cell slower than BF16. The SM90 Tensor Core path makes symmetric K8/V8 slower at B=1,T=32K, 1.09× faster at B=32,T=2K, and about 1.5× faster at B=32,T=32K. The same-family comparison is now closed: with split-dtype Tensor Core kernels compiled and audited at cosine 1.00000, symmetric K8/V8 beats K16/V8 by 16–19% at every bandwidth-bound shape (K8/V8 reaches 1.40–1.51× over BF16; K16/V8 reaches 1.19–1.28×), the K tax is negative at scale, and FP8 loses only at batch 1. On Blackwell's TensorRT-LLM generation kernels the transform modes converge and plain-E4M3 FP8 saturates at 1.05× over BF16, limited by its own pipeline rather than DRAM. The kernel family, not the storage format, decides the sign.
Use the exact H100 measurements
Keep the CUDA-core decomposition and the Tensor Core pair as separate experiments. They use different kernel families and therefore do not isolate scheduling by themselves.
FP8 loses on every tested cell
At B=32,T=32K cold, BF16 completes in 1,337.2 µs. K and V taxes are comparable, and quantizing both creates a +262.2 µs interaction.
The measured ordering crosses with batch
The same K8/V8 storage loses at B=1,T=32K and wins at B=32,T=2K. Run the complete B-by-T cross before ranking batch, occupancy, and amortization.
CUDA-core four-cell matrix
| shape | F16 | K8 | V8 / K16V8 | K8V8 | K tax | V tax | interaction |
|---|---|---|---|---|---|---|---|
| B=1 T=32768 warm | 56.8 | 65.3 | 67.3 | 75.8 | +8.5 | +10.5 | -0.0 |
| B=32 T=2048 warm | 99.1 | 107.1 | 108.1 | 141.9 | +8.0 | +9.0 | +25.7 |
| B=32 T=32768 warm | 1332.3 | 1461.0 | 1438.9 | 1826.4 | +128.7 | +106.6 | +258.8 |
| B=32 T=32768 cold | 1337.2 | 1469.1 | 1446.7 | 1840.8 | +131.9 | +109.5 | +262.2 |
Medians in microseconds. Treat this implementation as conversion- and schedule-bound at these points. Warm and cold B=32,T=32K results remain effectively identical.
SM90 Tensor Core symmetric pair
| shape | F16 µs | K8V8 µs | F16 / K8V8 |
|---|---|---|---|
| B=1 T=32768 cold | 43.1 | 53.9 | 0.80× |
| B=32 T=2048 cold | 59.7 | 55.0 | 1.09× |
| B=32 T=32768 warm | 712.8 | 471.1 | 1.51× |
| B=32 T=32768 cold | 712.6 | 478.6 | 1.49× |
Bound the exposed preparation cost
Use the pure transform probe as an upper bound on exposed conversion. Use the QK-consuming probe to measure how readily a consumer overlaps it.
Warm BF16 loads can beat the FP8 transform despite reading twice the bytes. Cold traffic eventually exposes the FP8 byte advantage at the tested large tile.
A minimal QK consumer hides most of the isolated transform tax. Measure the complete Tensor Core mainloop before making K preparation the dominant explanation.
Apply bounded conclusions
Record the exact kernel, JIT URI, cubin, layout, transform mode, and launch policy for every result.
K-only and V-only taxes are comparable. The K8V8 interaction becomes large at batch 32.
Keep conversion in the model, but measure the unhidden full-mainloop term.
Compare Full, KOnly, and SeparateKv with identical K8/V8 storage and one Tensor Core family.
Hold V, scales, pages, mask, output type, and launch policy fixed.
Run the full B-by-T cross before attributing the crossover.
Measure the tile dependencies
Advance online softmax tile by tile. Complete K preparation before that tile's QK. Complete V preparation before PV. Measure overlap for the selected kernel.
Run the decisive phase 2 campaign
Use a cheap B200 capability gate. Pivot to the H100 split-dtype template patch immediately when B200 cannot provide the same-family asymmetric and symmetric cells.
Preserve the phase 1 provenance
Read the archived H100 findings →
Inspect the phase 1 result commit →