Cartridges

Prefill speedup

Cartridge prefill time scales sub-linearly with prefix length because loading pre-computed KV (memcpy) is cheaper than running a forward pass over p tokens. Measured on Llama-3.2-1B-Instruct, AMD W7900, FlexAttention + torch.compile.

End-to-end throughput (tok/s)

At p=16,384: Cartridge achieves 44.5 tok/s vs ICL at 27.4 tok/s (1.6x end-to-end). The speedup comes entirely from prefill savings; decode throughput is identical.

vLLM TTFT on H100

On H100, prefill at 4K tokens takes only ~80µs of GPU time. TTFT is dominated by Python/HTTP overhead (~5ms), vLLM scheduling (~10ms), and network latency (~3ms). The cartridge advantage manifests at longer prefixes, lower-end GPUs, or higher concurrency.

Quality evaluation

LLM-as-judge: 10 factual questions per document, each answer scored 0–5. Instruct models self-judge; Llama-2-7b answers judged by Qwen-7B.

Qwen2.5-7B-Instruct

Key findings

Instruct tuning is critical. At p=256, instruct models score 2.4–4.0 while the base model scores 0.0–3.2 with high variance.

Model size matters less than expected. Qwen-1.5B (1.5B params) often matches Qwen-7B (7B params), suggesting cartridge training adapts well to smaller models.

Prefix Tuning

Cartridges build on Prefix Tuning (Li and Liang, 2021). The core idea: prepend p trainable continuous vectors to the key and value matrices at every attention layer. These vectors are optimized via gradient descent while the model weights stay frozen.

Initialization strategy matters

Li and Liang found that initialization strongly affects final quality:

First-k initialization (used by cartridges) is the natural extension: the initial KV state comes from actually processing the document, giving the optimizer a warm start in a region of KV space the model already understands. The optimizer then refines these vectors to compress the full document's information into p positions.

From fine-tuning to inference optimization

Original prefix tuning was a fine-tuning technique: train task-specific prefixes to adapt a frozen model. Cartridges repurpose it as an inference optimization: the trained prefix is the document representation, and injecting it replaces prefill entirely.

SCI vs First-k

SCI (Sampled Chunk Initialization) samples random 64-token chunks from across the document instead of taking the first p tokens. In theory, this captures broader content and may help optimization reach useful document coverage sooner. The original cartridge timing run showed similar fixed-budget wall-clock and decode behavior:

That table does not prove faster convergence. It reports fixed-budget runtime, not steps-to-quality or loss-to-threshold. We first tested SCI on SkillsBench skill files, but that is a skill-cache experiment, not a valid long-document cartridge test. The corrected July 2026 A100 document runs use LongHealth patient records with 9,603 to 12,503 wrapped source tokens and a 2,048-token learned KV prefix. Training data, objective, model, prefix size, and evaluation are held fixed; only the initializer changes.

The first corrected run on patient_01 showed a seed-specific SCI win: first-k reached 11 / 20, while three SCI seeds reached 10 / 20, 11 / 20, and 12 / 20. The follow-up A100 run expanded the test to four more LongHealth records:

Result: random paper SCI is not a robust faster-convergence default on these long-document cartridge tests. Across the four-patient run, SCI wins once, ties twice, and loses once, with a mean best-score delta of -0.5 correct answers. The useful next SCI variant is not "more random chunks"; it is a better chunk selector: retrieval-guided, strategy-guided, salience-guided, or section-balanced SCI.

The SkillsBench xlsx run remains useful, but it answers a different question: initializer behavior for skill caches induced from procedural skill files. That run also showed seed sensitivity, with first-k winning AR-TA quality and selected SCI seeds helping SS-PI. It should not be cited as document-cartridge evidence.

vLLM integration (v0.4)

CartridgeConnector is a KVConnectorBase_V1 plugin built as a production stack of seven modules (~2,800 lines). It supports single-cartridge (backward-compatible) and multi-cartridge (per-request dispatch) modes with no vLLM core modifications.

Production stack

Request lifecycle

Singleton mode (backward compatible)

vllm serve Qwen/Qwen2.5-7B-Instruct \
  --kv-transfer-config '{
    "kv_connector": "CartridgeConnector",
    "kv_connector_module_path":
      "vllm.distributed.kv_transfer.kv_connector.v1.cartridge_connector",
    "kv_connector_extra_config": {
      "cartridge_path": "/path/to/cartridge.pt"
    },
    "kv_role": "kv_both"
  }'

Multi-cartridge mode

vllm serve Qwen/Qwen2.5-7B-Instruct \
  --kv-transfer-config '{
    "kv_connector": "CartridgeConnector",
    "kv_connector_module_path":
      "vllm.distributed.kv_transfer.kv_connector.v1.cartridge_connector",
    "kv_connector_extra_config": {
      "cartridges": {
        "medical":  "/data/cartridges/medical.pt",
        "legal":    "/data/cartridges/legal.pt",
        "clinical": "/data/cartridges/clinical.pt"
      },
      "router": {"type": "label", "field": "cartridge_id"}
    },
    "kv_role": "kv_both"
  }'

Two concurrent requests can inject different cartridges into different allocated blocks without cross-contamination. The router carries cartridge_id end-to-end from the scheduler to the worker side of the connector.

W7900 validation

ROCm build profile

# Minimal build for cartridge-only serving on W7900 / gfx1100
VLLM_BUILD_PROFILE=cartridges_rocm pip install -e .

Builds only the extensions a cartridge-serving path needs (vllm._C, vllm._rocm_C, cumem_allocator). Includes a gfx1100 sampler thread-count fix and a gfx1x skinny_gemm wave32 workaround.

Cross-model analysis

Key findings

GQA models produce dramatically smaller cartridges. 8.5x smaller than MHA at the same prefix. GQA is strongly preferred.

Instruct tuning is critical for compression. Base Llama-2-7b scores 0.0–3.2 at 4x+ compression vs 2.4–4.1 for instruct models.

Small instruct models are surprisingly effective. Qwen-1.5B (1.5B params) often matches or exceeds Qwen-7B (7B params) at compressed cartridge sizes.

Concurrency scaling (H100)