knlp / serving & kv cache

knlp SPF

SPF is a scheduler-side speculative-prefetch experiment for vLLM's KV cache. It does not clear its own survival gate on Llama-3.2-3B under cache pressure. The branch is preserved so future scorers can be measured against the same bar.

What SPF tried

Predict which KV blocks a request will need before it arrives. Touch those blocks on the GPU block pool so they survive the next LRU eviction round (retention mode), or initiate async promotion from a slower tier (prefetch mode). The framework lives in vllm/v1/core/spf/ and is opt-in via VLLM_SPF_ENABLED=1; the scheduler bridge is a no-op when disabled.

Mechanism
retention
Promote cached blocks in the free queue via BlockPool.touch(). No allocation, no copy.
Scope
scheduler-side
Scores at request arrival; never touches attention, the forward path, or KV tensors.
Status
parked
Did not clear the survival gate. Branch preserved so a future scorer can be measured against the same arms.

The bar SPF was held to

The survival gate runs five arms through a real vLLM engine under cache pressure. The bar is what a sophisticated scorer must clear to live in the tree.

armpolicyrole
lruvLLM default prefix cachefloor
session_ttlsession-TTL + pinning of recently-active sessionstrivial baseline a sophisticated scorer must beat by a real margin
frequencyLFU-style admissionpolicy comparator
spfExpectedUtilityScorer (production)the candidate
oracleBelady reuse-distance, unimplementableceiling

A scorer earns its place in the tree iff, across at least three deterministic seeds:

  • p95 TTFT improvement vs session_ttl is at least 5%, OR
  • recomputed prefill tokens drop by at least 10% vs session_ttl,
  • throughput regression vs lru is under 1%,
  • wasted promoted bytes are under 10% of total promoted,
  • paired bootstrap CI on the headline metric excludes zero.

The result

Run the gate on Llama-3.2-3B-Instruct under 2.3x cache pressure (16 personas x 8192 token prefix each, 32 sessions x 4 turns, gpu_memory_utilization=0.3, max_model_len=12288, 3 seeds, L40S 48GB). The five arms land within 1% of each other on every measured axis.

arm p95 TTFT (ms) prefill recomputed cache hit throughput (rps)
lru772.901,064,9600.00%1.31
session_ttl778.451,063,3920.15%1.30
frequency779.531,063,3920.15%1.30
spf775.931,063,3920.15%1.30
oracle777.861,063,3920.15%1.30

SPF vs session_ttl: p95 TTFT improvement +0.32% (gate ≥5%), prefill reduction 0.00% (gate ≥10%). The bridge issues hints (430+ touches per run) and the controller is wired correctly; the policy does not produce a useful signal in this regime.

The oracle floor. Even the Belady reuse-distance oracle lands within 0.5% of session_ttl on round-robin pressure. The selector ceiling on this workload class sits below the gate. No learned scorer can rescue SPF on round-robin alone — a different workload pattern (skewed sessions, bursty access, same-session revisit) is required to give a scorer something to grip on.

What survives

Methodology
The survival gate

benchmarks/spf_survival_ab.py is the reproducible kill artifact. Any future scheduler-side admission or retention policy can be measured against the same arms.

Pattern
Disposal discipline

When an experiment cannot clear its own bar, delete the runtime, preserve a single archive commit and the benchmark, document the failure as a structural finding. Done with SPF.

Substrate
Manifest provider

BlockManifest + CartridgeKRIProvider were originally written for SPF consumption. The vLLM routing branch now consumes them as routing-prior loaders. See KRI and routing.

Reproduce

Check out the SPF branch, install vLLM precompiled, run the gate on a 48GB-class GPU (L40S, A6000, A100). One model, three seeds, five arms, about 90 minutes per seed on L40S.

install
git clone --branch 20260622-spf https://github.com/mcgrof/vllm.git
cd vllm
uv venv --python 3.12
VLLM_USE_PRECOMPILED=1 uv pip install -e . --torch-backend=auto
run the survival gate
.venv/bin/python benchmarks/spf_survival_ab.py \
    --model meta-llama/Llama-3.2-3B-Instruct \
    --arms lru,session_ttl,frequency,spf,oracle \
    --seeds 42,43,44 \
    --num-sessions 32 --turns-per-session 4 \
    --num-personas 16 \
    --shared-prefix-tokens 8192 \
    --per-query-tokens 128 \
    --output-tokens 32 \
    --gpu-memory-utilization 0.3 \
    --max-model-len 12288 \
    --enforce-eager \
    --output spf_survival_results.json

If you want to try

  1. Check out the SPF branch.
  2. Replace the scorer in vllm/v1/core/spf/scorer.py or add a new class and route it through SPFController._build_scorer.
  3. Run benchmarks/spf_survival_ab.py against your scorer.
  4. Clear the gate or document the next negative.

The bar exists so the line either delivers or stays parked. A plausible direction not yet tested: skewed-popularity or same-session-revisit workloads where session-recency tracking should diverge from session-frequency tracking.

Branch layout

github.com/mcgrof/vllm @ 20260622-spf

One commit on top of cart-upstream. The whole experiment.

  • vllm/v1/core/spf/controller.py
  • vllm/v1/core/spf/scorer.py
  • vllm/v1/core/spf/scheduler_integration.py
  • vllm/v1/core/spf/manifest.py
  • vllm/v1/core/spf/providers/cartridge_kri.py
  • vllm/v1/core/sched/scheduler.py # 4-line hook
  • tests/v1/core/test_spf_*.py # 18 files
  • benchmarks/spf_survival_ab.py

Related branches

The substrate that originated under SPF now lives on the routing branch as well.

  • 20260622-routing — serving substrate (null_block plumbing, routing-policy module, routing-prior algorithm hookpoint, xa refinement, native LMCache adapter). Independent of SPF.
  • 20260430-cartridges-upstream — the cartridges base both branches sit on.