Why coordinate shared inference state?

1. What redundant work do related requests repeat?

Agentic inference exposes at least two distinct forms of KV-cache redundancy, and it is worth keeping them separate. DualPath reports high cross-turn reuse in long-context, short-append agent trajectories — the same trajectory revisiting most of its own context turn after turn. TokenDance reports high cross-agent KV-block similarity after position-independent recovery in synchronized All-Gather rounds — sibling agents that share most of a round's state. The former is temporal reuse within a trajectory; the latter is sibling-state redundancy across concurrent requests. They are related architectural pressures, not the same measurement or reuse mechanism, and collapsing them into a single "shared cache" number obscures both.

What they share is the diagnosis. Request-centric execution usually flattens round, segment, branch, and lineage structure before it reaches the runtime. The serving engine sees token streams and has to rediscover relationships the agent orchestrator already knows — or misses them and recomputes state it could have reused. The opportunity is not that a clever cache would help; it is that the structure needed to exploit reuse safely is discarded at the door.

2. What do TokenDance and ForkKV show?

Two lines of work recover that structure deliberately, and both point past their own scope.

TokenDance lifts the unit of optimization from a single request to an agentic round. It is built from four separable mechanisms: round-aware segment indexing that preserves logical block boundaries even at different absolute positions; collective position-independent KV reuse that pays the recovery cost once per round instead of once per request; a Master–Mirror diff-aware storage scheme that encodes sibling caches as sparse differences against one dense master; and fused sparse restoration on transfer. KMesh implements an independent baseline of segment-aware collective recovery and exact family storage before testing broader abstractions. The bounded baseline below covers selected mechanisms; it is not a reproduction of the paper's complete serving system.

ForkKV shows the opportunity is larger than the KV cache alone. Multi-LoRA agents co-hosted on one base model diverge in their KV because each adapter changes the activations for the same context, which defeats naive prefix caching. ForkKV borrows process semantics directly — fork the massive shared cache and apply copy-on-write for the lightweight, agent-specific remainder, tracked by a radix tree and reconstructed on chip. The lesson is that the shareable unit is the keyed inference state: the KV, the adapter-affected activations, and the prefixes they hang on.

3. Can the shared object extend beyond KV cache?

The same shared-trunk, per-branch-diff structure is not agentic-specific. Chain-of-thought reasoning revisits and branches its own context; test-time scaling forks a common prefix into parallel samples, best-of-N, or tree search; speculative decoding branches and prunes. These are the same pattern that TokenDance and ForkKV exploit for agents, and there is no reason the machinery should stay agent-specific.

KMesh investigates whether an inference engine — vLLM first — can expose explicit, keyed state relationships that let recovery, scheduling, storage, and later memory tiering operate on groups of related requests. The unit is the keyed inference state: a per-context artifact keyed by the context it derives from, so a workload that provably shares that context can share, fork, or diff the state instead of recomputing it. Agentic serving, chain-of-thought, and test-time scaling then become callers of one mechanism rather than three separate optimizations. Whether that unification holds is a question to answer with an implementation, not to assume.

4. Which parts of the architecture remain unresolved?

The long-term architecture is intentionally unresolved, and saying so is the point. Linux virtual-memory primitives, accelerator memory, CXL tiers, and a generalized inference-object model may eventually matter — but those are hypotheses, and announcing that the operating system needs new interfaces before an inference engine has shown which mechanism is actually missing puts the cathedral before the door. The immediate work is empirical: use a real vLLM implementation to identify the smallest reusable abstractions and the precise mechanisms current runtimes lack.

One interface question is already concrete. TokenDance carries segment structure with a separator token in the prompt; that is a fine paper-compatibility mode, but a reasoning or sampling orchestrator already knows its own branch and segment boundaries and should not have to inject control tokens into a model prompt to tell the runtime about them. So the mechanism should consume explicit segment metadata, with the in-prompt separator as one encoding of it rather than the internal contract. Getting that boundary right is what keeps a TokenDance reproduction from foreclosing chain-of-thought support later.

5. What does the implementation test?

The implementation separates collective recovery, physical prefix sharing, attention computation, external restoration and future-use coordination. The TokenDance-style baseline checks segment-aware recovery and Master–Mirror storage under controlled All-Gather workloads. The serving, DMA-BUF, singleflight and family experiments then test their own mechanisms with explicit controls.

Exact-prefix forks and family declarations test the extension toward chain-of-thought and other test-time-scaling branches. Results on a single workstation-class GPU are labeled as functional or trend results; a comparison against the paper's numbers waits for comparable data-center-class hardware, with every material difference disclosed. The standard is not that the code runs — it is that the workload is controlled, the mechanism is observable, the comparison is fair, the fallback is safe, and the claim says no more than the evidence.