Routing decides which KV blocks each request attends to. A serving substrate hosts the algorithm; the algorithm decides the blocks. Pick a different algorithm and the substrate does not change. Work with cartridges if you want fast experimentation; the substrate does not require them.
Read routing as two layers with a small contract between them. The serving substrate in vLLM owns scheduler plumbing, block selection, sparse inject, and an optional cross-attention refinement side-channel. The algorithm answers a single question: for this request, which K of the cartridge's blocks should each KV head see? Today the algorithm slot hosts KRI variants and policy-based selectors (hybrid, recency). Tomorrow it hosts anything that exposes the same interface.
CartridgeConnector with routing-aware
sparse inject, scheduler null_block plumbing, routing-policy
module, thread-local routing state, xa refinement, native LMCache
adapter. Six clean commits.
BlockManifestProvider
plugs in. The connector resolves (prefix_hash, query_hash) → BlockManifest
at request time.
Read the algorithm slot as two orthogonal questions, not one. Algorithms that answer different questions compose; algorithms that answer the same question compete. Keep them distinct in your head and in the configuration.
Decide which subset of the cartridge's blocks each request actually
attends to. The remaining blocks become
null_block; the attention backend
skips them. Selection determines the shape of the routed cache.
After the selected blocks are injected into paged cache, identify positions whose K or V have drifted relative to a fresh forward and replace them. Repair fires once per request after inject and before the first decode step.
CacheBlend (Yao et al, 2024 — arxiv 2405.16444) observes that, after reusing a cached KV from one context inside a different surrounding context, a small fraction of positions account for most of the quality loss. CacheBlend identifies those positions by their Highest KV Deviation (HKVD) against a fresh reference forward, then recomputes K and V at exactly those positions before generation continues. The paper shows cross-layer correlation of HKVD scores is high (Spearman ≈ 0.9 on Mistral / Yi / Llama), so the deviation pattern measured in one layer transfers to the rest.
xa25 is the production parameterization knlp ships: fixed HKVD refresh ratio = 25%, single-pass (the iterative escalation from the paper is cut), reference forward run once per request. The naming — x-attention, across-context, 25% — ties the parameter into the identifier so the configuration shows up in eval logs without a separate column.
Important. xa25 amplifies a good selector. It lifts KRI-D + structural pinning by +1.5 to +3.5pp on LongHealth K=16; it stays flat on random middle (−0.5pp). Repair is not a free lift — it cannot rescue a selector that picks the wrong blocks.
Each algorithm produces a per-block score offline and writes it as a
.pt prior keyed by the cartridge's prefix
hash. At serve time the connector loads the prior through a
BlockManifestProvider and resolves it
to a top-K selection per request.
| algorithm | signal | query-aware | cost | status |
|---|---|---|---|---|
| KRI-D-kv-sum [knlp-authored] | Sum over layers of L2 between full-cache hidden and leave-one-block-out hidden. L2M-inspired — L2M (arxiv 2503.04725) is an information-theoretic scaling law for long-context modeling, not a KV-cache method; KRI-D applies its bipartite-MI framing to per-block importance, using hidden-state L2 as the empirical proxy. | no | offline; one prefill per block | production leader: K=16 + xa25 = 26.5% on LongHealth (5σ vs random 21.5%, n=200) |
| KRI-D [knlp-authored] | KL divergence of logits, leave-one-block-out. L2M-inspired — the original KRI-D framing: per-block instantiation of the L2M sum rule (arxiv 2503.04725), with the model itself as the bipartite-MI estimator. | no | offline; one prefill per block | live; weaker than kv-sum variant |
| KRI-A [knlp-authored] | Prefill attention probability mass per block, query-agnostic variant. Heavy-Hitter family: H2O Heavy-Hitter Oracle (arxiv 2306.14048) — training-free at inference in the original paper. | no | one prefill | live baseline; recency bias |
| KRI-G [knlp-authored] | Gradient of LM-head loss w.r.t. block-mean K. No direct external ancestor. | no | one backward pass | live |
| KRI-Q [knlp-authored] | Q·K agreement against block-mean K. Query-aware family: Quest (arxiv 2406.10774) — training-free at inference in the original paper. | yes | cheap online | live; flat on untrained K means |
| KRI-T [external] | Pre-RoPE Q/K trigonometric factorization. Direct implementation of TriAttention (Mao et al, arxiv 2604.04921). | yes | cheap online | live; needs multi-needle eval |
| kmeans_keys [knlp-authored] | K-means cluster centroids over block-mean K. Borrows the block-clustering gating signal from MoBA (arxiv 2502.13189). Note: MoBA's original method requires training the gate; knlp evaluates the signal training-free, applied to an off-the-shelf checkpoint at inference time. | no | one prefill + k-means | live |
| attention_prefill [knlp-authored] | Layer-mean prefill attention (post-softmax). Cousin of SnapKV's prefill-attention compression (arxiv 2404.14469) — training-free at inference in the original paper. | no | one prefill | live |
| eigenvector centrality [knlp-authored] | Dominant eigenvector of the block-level prefill attention graph. No direct external ancestor. | no | one prefill + power iteration | live; addresses KRI-A recency bias and KRI-D marginal-MI redundancy |
| hybrid (anchor + recency) [external] | First K_anchor blocks + last K_recent blocks; middle dropped. Attention-sinks construction from StreamingLLM (Xiao et al, arxiv 2309.17453). | no | none (no prior baked) | live; serves as cheap policy comparator |
| recency [external] | Last K blocks. Sliding window from StreamingLLM (arxiv 2309.17453). | no | none | live; floor for end-weighted tasks |
| xa25 (CacheBlend overlay) [external] | Post-load refresh of top-25% HKVD positions with fresh K/V. Parameterized at 25% HKVD refresh rate; single-pass. From CacheBlend (Liu et al, arxiv 2405.16444). | yes (via reference forward) | one extra forward | repair overlay; lifts KRI-D + A1R2K13 by +1.5 to +3.5pp on LongHealth K=16 |
Read the production stack as: KRI-D-kv-sum for selection, A1R2K13 structural pinning (anchor=1, recent=2, KRI-mid=13) at K=16, xa25 as the post-load repair overlay. Validated on LongHealth K=16: 26.5% vs random middle 21.5%, 5σ at n=200.
The routing branch sits on top of cart-upstream and adds six commits
shaped around the substitution points. The CartridgeConnector consults
a manifest provider per request, applies the resolved selection to
its block table, and writes only the selected blocks into paged
cache through a Triton kernel; the rest stay
null_block and the attention backend
skips them.
request arrives
↓
CartridgeConnector.get_num_new_matched_tokens
↓ (routing config present?)
update_state_after_alloc
↓ resolve (prefix_hash, query_hash) → BlockManifest via routing_prior/
↓ store per-request selection set
get_block_skip_list
↓ complement of selection within [0, num_logical_blocks)
KVCacheManager.null_block_positions
↓ mutate block_table; free the freed physical blocks
build_connector_meta → CartridgeReqMeta carries the selection
↓
_inject_request
↓ triton_reshape_and_cache_flash on selected blocks only
_publish_routing_state
↓ thread-local RoutingPrior for the routed-decode backend
_maybe_xa_refine
↓ (opt-in) scatter fresh HKVD K/V into paged cache
forward pass: attention sees null_block at non-selected positions, skips them
| substitution point | where | what you swap to try a new idea |
|---|---|---|
| Algorithm | routing_prior/ |
Implement BlockManifestProvider.get_manifest(prefix_hash, query_hash, K); drop your .pt priors in |
| Block-selection policy | routing_policy/ |
Add a function that returns a list of block indices; the connector calls it directly when no prior is configured |
| Scheduler hook | KVConnectorBase.get_block_skip_list |
Already generic; every connector can drive null_block. New algorithms do not touch the scheduler |
| Repair overlay | xa_refinement.py |
Same side-channel shape would fit CacheBlend-style iterative escalation or any other post-load repair mechanism |
| Storage backend | lmcache_connector.py use_native switch |
Native adapter with sparse retrieve_chunks; room for additional vendored backends |
Build the routing branch precompiled and run the existing cartridge test suite (198 tests) to confirm the substrate works on your box. Bake a KRI-D-kv-sum prior offline, register it under a cartridge id, and run the connector smoke tests with routing enabled.
git clone --branch 20260622-routing https://github.com/mcgrof/vllm.git cd vllm uv venv --python 3.12 VLLM_USE_PRECOMPILED=1 uv pip install -e . --torch-backend=auto .venv/bin/python -m pytest tests/v1/core/test_cartridge_*.py -q
--kv-transfer-config '{
"kv_connector": "CartridgeConnector",
"kv_connector_module_path":
"vllm.distributed.kv_transfer.kv_connector.v1.cartridge_connector",
"kv_connector_extra_config": {
"cartridges": {"patient_00": "/path/to/cart.pt"},
"routing": {
"enabled": true,
"K": 16,
"priors": {"patient_00": "/path/to/kri_d_prior.pt"}
}
},
"kv_role": "kv_both"
}'
Six commits on top of cart-upstream. Independent of SPF; sits directly on the cartridges base.
Prior bakery and KRI variants live in knlp. Each algorithm reduces
to a .pt file the connector loads.