SPF is a scheduler-side speculative-prefetch experiment for vLLM's KV cache. It does not clear its own survival gate on Llama-3.2-3B under cache pressure. The branch is preserved so future scorers can be measured against the same bar.
Predict which KV blocks a request will need before it arrives. Touch
those blocks on the GPU block pool so they survive the next LRU
eviction round (retention mode), or initiate async promotion from a
slower tier (prefetch mode). The framework lives in
vllm/v1/core/spf/ and is opt-in via
VLLM_SPF_ENABLED=1; the scheduler bridge
is a no-op when disabled.
BlockPool.touch(). No allocation, no copy.
The survival gate runs five arms through a real vLLM engine under cache pressure. The bar is what a sophisticated scorer must clear to live in the tree.
| arm | policy | role |
|---|---|---|
lru | vLLM default prefix cache | floor |
session_ttl | session-TTL + pinning of recently-active sessions | trivial baseline a sophisticated scorer must beat by a real margin |
frequency | LFU-style admission | policy comparator |
spf | ExpectedUtilityScorer (production) | the candidate |
oracle | Belady reuse-distance, unimplementable | ceiling |
A scorer earns its place in the tree iff, across at least three deterministic seeds:
session_ttl is at least 5%, ORsession_ttl,lru is under 1%,
Run the gate on Llama-3.2-3B-Instruct under 2.3x cache pressure
(16 personas x 8192 token prefix each, 32 sessions x 4 turns,
gpu_memory_utilization=0.3,
max_model_len=12288, 3 seeds, L40S 48GB).
The five arms land within 1% of each other on every measured axis.
| arm | p95 TTFT (ms) | prefill recomputed | cache hit | throughput (rps) |
|---|---|---|---|---|
lru | 772.90 | 1,064,960 | 0.00% | 1.31 |
session_ttl | 778.45 | 1,063,392 | 0.15% | 1.30 |
frequency | 779.53 | 1,063,392 | 0.15% | 1.30 |
spf | 775.93 | 1,063,392 | 0.15% | 1.30 |
oracle | 777.86 | 1,063,392 | 0.15% | 1.30 |
SPF vs session_ttl: p95 TTFT improvement
+0.32% (gate ≥5%), prefill reduction
0.00% (gate ≥10%). The bridge issues
hints (430+ touches per run) and the controller is wired correctly;
the policy does not produce a useful signal in this regime.
session_ttl on round-robin pressure. The
selector ceiling on this workload class sits below the gate. No
learned scorer can rescue SPF on round-robin alone — a different
workload pattern (skewed sessions, bursty access, same-session
revisit) is required to give a scorer something to grip on.
benchmarks/spf_survival_ab.py is the
reproducible kill artifact. Any future scheduler-side admission
or retention policy can be measured against the same arms.
When an experiment cannot clear its own bar, delete the runtime, preserve a single archive commit and the benchmark, document the failure as a structural finding. Done with SPF.
Check out the SPF branch, install vLLM precompiled, run the gate on a 48GB-class GPU (L40S, A6000, A100). One model, three seeds, five arms, about 90 minutes per seed on L40S.
git clone --branch 20260622-spf https://github.com/mcgrof/vllm.git cd vllm uv venv --python 3.12 VLLM_USE_PRECOMPILED=1 uv pip install -e . --torch-backend=auto
.venv/bin/python benchmarks/spf_survival_ab.py \
--model meta-llama/Llama-3.2-3B-Instruct \
--arms lru,session_ttl,frequency,spf,oracle \
--seeds 42,43,44 \
--num-sessions 32 --turns-per-session 4 \
--num-personas 16 \
--shared-prefix-tokens 8192 \
--per-query-tokens 128 \
--output-tokens 32 \
--gpu-memory-utilization 0.3 \
--max-model-len 12288 \
--enforce-eager \
--output spf_survival_results.json
vllm/v1/core/spf/scorer.py or add a new
class and route it through
SPFController._build_scorer.benchmarks/spf_survival_ab.py
against your scorer.The bar exists so the line either delivers or stays parked. A plausible direction not yet tested: skewed-popularity or same-session-revisit workloads where session-recency tracking should diverge from session-frequency tracking.
One commit on top of cart-upstream. The whole experiment.
The substrate that originated under SPF now lives on the routing branch as well.