Concepts · interactive foundations
Why does decode slow down? When do fewer bytes help? What does a correlation actually show? These guides connect the mechanics to the measurements.
Read in order, or jump to the question you have. Each guide includes a model or an interactive view you can explore.
Why does generating one more token mean reading so much old state?
Follow the causal decode loop, count the key/value reads, and vary history and generation length. This isolates attention-side traffic; whole-model latency also includes weights and other work.
Is the kernel waiting for bytes, or running out of compute?
Connect arithmetic intensity to memory bandwidth and compute throughput. Move between the two ceilings and see why a spec-sheet roofline differs from sustained performance.
What changes when batch, context and cache precision change?
Compare the page’s decode-scaling data across GPUs and workloads. Use batch, context and quantization views to connect a traffic model to observed behavior.
Do two quantities move together, and what can that tell us?
Build Pearson r, R² and Spearman ρ from the data. Change points, examine ranks, and see how outliers or nonlinear relationships alter the result.
See these ideas meet model behavior, cache precision and the storage path.
Mechanisms, causal controls and recovery options for fragile KV-cache layouts.
How cache capacity, kernel speed and batch size interact in decode throughput.
Early Linux DMA-BUF enablement, platform qualification and application tests.