What does real agent traffic ask of shared state?
The earlier questions measured sharing, retention and early loading on request families we built. This page asks what released agent traces actually contain: how often one agent starts many children at once, what those children share, and how long large contexts sit idle. It also records why vLLM removed cascade attention, the shared-prefix attention method that wide request families would use.
26. Which agent traces can answer these questions?
Four released traces, and each answers a different part. None of them releases prompt text. Two release block identifiers: each request's prompt is cut into fixed-size blocks, and equal blocks get equal identifiers. Equal leading identifiers then stand in for a shared prefix. That is a workload intention, not a verified KV identity on any model.
| Trace | Coverage | What it records | What it cannot show |
|---|---|---|---|
LMCache agent-trace corpus, revision 780bcc29 | 78 JSONL files, 1,915 unique records across agent frameworks | Record text statistics and timestamps | No parent links and no invocation timing; see question 17 |
Weka Claude Code corpus, revision 23f152f6, Apache-2.0 | 393 Claude Code sessions, 98,827 model requests, 1,697 subagents | Each request's input and output size, start time and duration; each subagent's spawn time and requests; 64-token hash blocks | Block identifiers are local to a session. Sampling kept sessions with at most 10 concurrent subagent groups, which removes the widest sessions |
FreeInference agentic trace, revision 16d3c04d, CC BY 4.0 | Weeks of 2026-05-17 (125 sessions) and 2026-07-12 (969 sessions), eleven harnesses including Claude Code, Codex, opencode and Kilo | Parent session and spawning step, 16-token block identifiers shared across the week, cache-read tokens | Few families: 17 and 146 launches |
GitHub Copilot coding agent traces, release ghcp-coding-agent-2026, CC BY | Seven days released; we read 2026-06-02 and 2026-06-06: 70,819 sessions, 276,539 turns, 2.18 million model calls | Each call's prompt, completion and cached tokens, model label and timing; tool names | No block identifiers and no subagent tool: the release is the Visual Studio slice |
The Copilot release accompanies Liu et al., Agentic Coding in the Wild, which studies a larger population of 13 million sessions. Its main observations for KMesh are that cache reuse is high inside a turn and much lower across turn boundaries, that model switches and context compaction invalidate reuse, and that users leave minutes-long idle gaps between turns. Our counts below come from the released days only.
One reading note for that release. An LLM call's timestamp
marks the end of the call. In 81,519 consecutive call
pairs, 99.94% fit that reading and 75% fit the start reading that the
release's own analysis code uses. Read as start times, calls appear to
overlap. Read as end times, two or more calls overlap in under 0.2% of
turns, and four or more in 0.02–0.06%. Tool-batch timestamps do not agree
with the call clock, so our turn times use model calls only.
27. Do agent launches fan out over a shared saved ancestor?
No. Wide launches exist, but their children share only the harness preamble. A launch is one parent step that starts one or more children; its width is the number of children. For each launch we compare the children's first requests. The group span is the prefix common to all of them. The preamble span is the part the first child also shares with a child of a different launch: the system prompt and tool definitions that every launch carries. The saved family ancestor is the part of the group span that existed before the launch and is not preamble. It is the state a storage read could serve to all children at once.
| Trace | Launches | Launches of 4+ children (their children) | Group span, p50 | Preamble span, p50 | Saved family ancestor, largest |
|---|---|---|---|---|---|
| Weka, spawns within 1 s | 1,456 | 29 (143 of 1,697) | 30.0K tokens | 30.0K tokens | 0 |
| Weka, spawns within 5 s | 1,141 | 75 (375 of 1,697) | 31.5K | 31.4K | 0 |
| FreeInference, July week | 146 | 22 (99 of 270) | 9.1K | 8.5K | 0 |
| FreeInference, May week | 17 | 0 | none | none | none |
Wide launches are 2% of Weka launches, with 8% of its children, and 15% of FreeInference July launches. They are four to seven children wide. What the children share beyond the preamble is at most 2.4K tokens in Weka at 1 s grouping, 21.8K at 5 s and 5.3K in FreeInference, and all of it first appears at the launch. Storage holds no copy of state that was only just produced. Children also receive almost nothing from their parent: in Weka, 1,374 of 1,697 children share no leading block with the parent's last request, and the largest share is 9% of the child's prompt. No harness in either trace hands its children the parent's conversation.
The preamble can go cold, but the repeated reads are small
The preamble is specific to a session, not shared by every user of a harness. In FreeInference, where block identifiers are shared across the week, a child shares a median of only 144 tokens with children launched by other users. A session's preamble can therefore leave GPU memory between its launches. A wide launch would then read it back once per child, which is the case singleflight removes. We counted it after the first comparison came back empty:
| Trace | Launches, idle 60 s or more | Their children | Repeated tokens a per-child read would load |
|---|---|---|---|
| Weka, spawns within 1 s | 13 (0.9% of launches) | 68 (4.0%) | 1.5 million |
| Weka, spawns within 1 s, idle 300 s or more | 6 (0.4%) | 30 (1.8%) | 0.65 million |
| Weka, spawns within 5 s | 32 (2.8%) | 162 (9.5%) | 3.4 million |
| FreeInference, July week | 1 (0.7%) | 4 (1.5%) | 75 thousand |
Singleflight only helps reads that are in flight together. In Weka, the children of a wide launch grouped at 1 s start within 0.2 s of each other at the median (0.7 s at p90). Grouping at 5 s merges spawns 2.8 s apart at the median and 12 s at p90, long enough for the first child's restore to finish before the next child asks. The 1 s row is the relevant one, and its 1.5 million repeated tokens compare with 21.6 billion input tokens in the whole corpus.
28. Where does long context sit idle?
Between turns. The resumed turn is the large storage target. In the Weka corpus we counted main-agent requests of at least 32K input tokens that arrive at least 60 s after the previous main-agent request ended. A resume counts as a subagent wait when a subagent was spawned inside the gap.
| Weka work | Count | Input tokens | Share of main-agent input tokens |
|---|---|---|---|
| Resume after a user or tool wait | 7,837 requests | 2.79 billion | 15.7% |
| Resume after a subagent wait | 445 requests | 122 million | 0.68% |
| Cold preamble re-reads in wide launches | 68 children | 1.5 million | 0.008% |
A parent that waits for a subagent is a large object with some notice: it resumes with a median of 229K input tokens (541K at p90) after a median wait of 205 s, and the subagent's recorded end comes a median of 8.6 s before the parent's next request. The user's own turn is about 23 times larger in total tokens, and it gives no notice.
The Copilot release shows what a production deployment does at that boundary. The cached-token fraction at a turn's first call is the deployment's own count. It falls off a cliff at five idle minutes:
| Idle before the turn | Weekday turns | Cached, p50 / mean | Weekend turns | Cached, p50 / mean |
|---|---|---|---|---|
| 10 to 60 s | 38,724 | 0.95 / 0.73 | 7,454 | 0.95 / 0.73 |
| 1 to 5 min | 68,027 | 0.94 / 0.71 | 12,617 | 0.95 / 0.72 |
| 5 to 30 min | 43,169 | 0.00 / 0.21 | 7,028 | 0.00 / 0.31 |
| 30 to 60 min | 7,355 | 0.00 / 0.14 | 991 | 0.00 / 0.25 |
| 1 to 24 h | 10,589 | 0.00 / 0.07 | 1,422 | 0.00 / 0.12 |
| Measure | Weekday, 2026-06-02 | Weekend, 2026-06-06 |
|---|---|---|
| Idle 60 s or more: share of turn boundaries | 51.8% | 48.7% |
| Idle 60 s or more: share of all prompt tokens | 7.1% | 5.8% |
| Idle 60 s or more: share of all recomputed prompt tokens | 26.3% | 23.5% |
| Idle 300 s or more: share of all recomputed prompt tokens | 19.2% | 16.8% |
| Idle 300 s or more: recomputed part of their own prompts | 82% | 75% |
| Resuming prompt, p50 / p90 | 80K / 164K tokens | 81K / 165K tokens |
| Idle gap, p50 / p90 | 274 s / 2,592 s | 240 s / 1,855 s |
About a fifth of all the prompt tokens this deployment recomputes belong to long-context turns that resume on the same model after five or more idle minutes. Storing a context at the end of a turn and restoring it when the next turn starts targets exactly those tokens. Unlike a subagent wait, the user's return gives no notice, and the gap runs from minutes to an hour, so the stored copy must stay in storage that long. A model switch (about 5% of turn boundaries) or a compaction (about 3%) invalidates a stored copy; the table excludes both.
29. What happened to cascade attention in vLLM?
vLLM removed it in October 2026, after it had been off by default since March. Cascade attention computes attention over a prefix shared by many requests once, computes each request's private part separately, and merges the results. Question 12 enabled it explicitly with an 8,192-token shared ancestor and 32 children: generation wall time fell 17.8%, and 24 of 96 continuations changed because the floating-point evaluation order changed.
| Date | Change |
|---|---|
| 2025-11-21 | #25266 adds a rewritten GPU model runner. It never implements cascade. |
| 2026-03-11 | #36318 turns cascade off by default "to avoid potential numerical issues by default". It changes only the configuration default. |
| 2026-08-27 | #53183 makes the rewritten runner the default for all models. |
| 2026-10-12 | #61126 removes cascade attention: kernels, heuristics, scheduler and KV-cache plumbing, runner logic, tests and documents. It first ships in v0.32.0. The disable_cascade_attn option remains as a no-op that is scheduled for removal. |
The removal states its reason directly. Cascade attention "has been
opt-in (disable_cascade_attn=True by default), already
hard-disabled for FlashInfer (see #25679), force-disabled for async spec
decode / DBO /
batch invariance / CPU, and unsupported by most backends — yet it still
costs a per-step scheduler scan, branches in the model runner and backends,
and recurring bugs." Its tests check accuracy and identical outputs on a
shared-prefix batch; it reports no performance comparison.
Why it reached so few batches
The engagement rule was strict. Before each step, the scheduler counted the leading KV blocks referenced by every request holding KV in the engine, not only the requests in the batch. One request without the prefix turned cascade off for the whole batch. The code's own comment notes that the count can be zero "even though all scheduled requests share a common prefix". There was one shared prefix per batch, with no support for several request groups that each share a different prefix. The reference count could also be wrong: issue #61673 reports a request decoded against another request's KV prefix while a CPU offload connector was in use.
Other serving stacks are in a similar place. SGLang's cascade kernel request, #1715, has been open since October 2024. TensorRT-LLM added a cascade kernel for beam-search decode in #14396, behind an environment variable that is off by default. The published gains come from batches built to share a prefix, beam or tree decoding, or a common system prompt: Hydragen, FlashInfer's cascade inference, Bifurcated Attention, DeFT and PAT. PAT groups requests by shared prefix as a vLLM plugin and reports 17–93% lower time per output token. We found no evaluation on Claude Code, Copilot or SWE-agent traffic.
Was it dropped because agent workloads have no use for it?
Partly, but that is not the stated cause. In order of evidence strength:
- Engineering scope. The rewritten runner never had cascade, and it became the default for all models. The removal cites maintenance cost against little reach. This is stated upstream.
- The engagement rule. The whole engine had to share one prefix, which mixed serving traffic rarely does. This is in the source.
- The workloads. The traces in questions 27 and 28 show agent launches four to seven wide that share a session preamble. Cascade pays when many concurrent decodes read one long prefix: reinforcement learning rollouts with many samples per prompt, best-of-n sampling, beam search and tree search. This is our inference from the traces and the published results.
- Numerics. Changed outputs explain the off-by-default change in March, not the removal.
The kernel still works where wide same-prefix groups exist, as question 12 measured. A return of cascade to vLLM would need a per-group design in the current runner and a workload with such groups. On this evidence, agent serving is not that workload.
What the evidence points to
In agent traffic, the KV cache worth saving is each session's own context between turns, not context shared between agents. KMesh's work moves to that resumed turn: which contexts to keep, in which memory tier and for how long, so that a resumed turn restores its context instead of computing it again. Singleflight stays in place for the rare wide launch; the traces give it little volume. Early preparation still matters for subagent waits, which come with seconds of notice, but they are a small share of the idle tokens. The overview lists what continues and what is parked.
These are workload counts, not KV measurements. Equal block identifiers are a workload intention. Weka's sampling removes the widest sessions, so a corpus of heavy parallel exploration, such as tree search or many-agent planners, could still contain wide launches over a large saved ancestor. FreeInference's family counts are small. Copilot's subagent behavior in its other editors is not in the public slice. The thresholds of 16K tokens, 60 s and 300 s are our choices, not measured break-even points.
Reproduce the trace counts
cd tools/reproduce/kmesh_traces
python3 fanout_spans.py --weka /path/to/weka/traces.jsonl --window-s 1 --output weka-w1.json
PYTHONPATH=/path/to/fi-artifact python3 fanout_spans.py \
--release /path/to/fi-layout --week 2026-07-12 --output fi-2026-07-12.json
python3 weka_resume.py --traces /path/to/weka/traces.jsonl --output weka-resume.json
python3 copilot_resume.py --data /path/to/copilot --days 2026-06-02 --output copilot-2026-06-02.json
The trace README gives the pinned revisions, download commands, archive digests, the FreeInference reader layout and the span definitions. Everything runs on a CPU from published data; no model or GPU is needed.