Upstream dependencies
and the LMCache series
vLLM upstream integration waits for the FlashInfer asymmetric K/V capabilities it uses to merge. LMCache has already merged the split serialization/deserialization foundation; multi-process storage support follows in a separate series. The current tested inference recipe uses pinned public forks and does not require LMCache.
Status checked October 19, 2026. A public feature branch, an open pull request and a merged change are different states.
Serving: FlashInfer enables vLLM integration
FlashInfer PR #3812 is open and unmerged. It adds separate K/V cache dtypes for CUDA-core batch decode: 16-bit keys and FP8 values. It is one part of the serving support; it does not by itself upstream the full measured tensor-core and prefill path.
The current tested sources are FlashInfer 800a550 and vLLM 276305b. The vLLM integration follows the relevant FlashInfer merges; it does not need unrelated FlashInfer changes to land first. LMCache storage review can proceed in parallel.
LMCache: split SerDes first, MP-mode storage next
SerDes means serialization and deserialization. PR #3277, merged June 16, supplies the CPU-side multi-output interface and asymmetric K/V codec. That allows key and value data to be represented separately. It does not complete the multi-process (MP) storage path that places those components on different tiers.
Issue #3710 tracks that next step. Its current submission map starts with general correctness fixes, followed by three feature branches. The feature labels PR0, PR1 and PR3 identify planned slices, not numbered upstream pull requests. There is no separate PR2 in this revised series.
| Stage | What it adds | Current status and code |
|---|---|---|
| Foundation | Multi-output K/V SerDes and the asymmetric codec. | #3277 merged |
| Prerequisite fixes · 7 commits | Safe transfer ownership and storage contracts in existing LMCache paths. | #5502 open · branch · fixes only |
| Typed dispatch · PR0 · 3 commits | Route separate K/V outputs through MP L2 adapters, with component layouts and actual serialized lengths. | Published branch; queued after fixes · branch · PR0 only |
| Split-tier placement · PR1 · 7 commits | Keep K in CPU DRAM L1 and V in filesystem L2. Require both children for a cache hit; pair eviction and guard in-flight work. | Published branch; follows PR0 · branch · PR1 only |
| FP8 byte-through · PR3 · 3 commits | Preserve existing FP8 value codes without a full-precision round trip. Add RAW_UNIT/V2 format, provenance checks and key separation. | Published branch; follows PR1 · branch · PR3 only |
The three feature branches are prepared for review; they are not merged MP support. The reference base is 8c77a6f77, and the four slices contain 7, 3, 7 and 3 commits respectively: 20 in the combined stack. Standalone fix-topic branches are reference copies, not additional PRs.
A parallel contribution, PR #4504, is an open multi-output MP contract-test proposal. It tests the codec/async processor boundary and records missing placement wiring as an expected failure. It is test coverage, not the implementation of the placement series.
The seven fixes in PR #5502
These repair existing shared paths and can benefit users who do not use asymmetric quantization. The central rule is that a timeout, failed heartbeat or exception does not prove an asynchronous copy has stopped. Its buffers must stay owned until completion is established.
| Fix | Failure it prevents | Commit |
|---|---|---|
| Retain retrieve sources after partial submission | Earlier host-to-device copies reading buffers released after a later submission error. | 5cb27221 |
| Retain submitted transfers across health loss | Pending work and IPC resources discarded while remote GPU work may still use them. | cf093fab |
| Require proven store completion | Preemption or shutdown reusing KV blocks still referenced by a transfer. | 5cf31f00 |
| Drain filesystem I/O before release | Delayed unlink removing replacement files, or delayed reads writing into reused buffers. | 0dc0b7d2 |
| Refresh paged-allocation offsets | Recycled objects retaining byte offsets from an incompatible prior layout. | 9de30b8c |
| Reject negative encoded section lengths | Invalid K, V or scale boundaries accepted despite a matching payload size and checksum. | 0863f22d |
| Batch stores by object group | One store batch mixing objects with different tensor layouts. | 82470898 |
What the MP-mode series establishes
The admitted split-tier configuration uses CPU DRAM L1, one filesystem L2 adapter, one object group and one K/V pair per object. It composes with the existing asynchronous write-behind path. K and the manifest are process-local: value-only files cannot reconstruct a split-tier cache hit after restart.
The October 5 report records 966 CPU tests passed with 45 skipped for the seven-fix boundary, and 2,392 passed with 147 skipped for the combined stack. The selections overlap and must not be added. The refresh is not a new exact-stack GPU run or full fused-serving qualification. Earlier GPU tests retain their original scope.
Full byte-through serving needs a matching vLLM/FlashInfer producer and the codec’s scale/provenance contract. Ownership-preserving fixes can block preemption or shutdown when completion cannot be proven; a real two-process CUDA/vLLM restart test under in-flight reuse pressure remains a follow-up in the issue. Reduced stored bytes alone do not establish throughput, latency, quality or SSD-endurance gains.
Read the maintainer-facing submission map and qualification record · Return to the inference guide.