Cartridges

Prefill speedup

Cartridge prefill time scales sub-linearly with prefix length because loading pre-computed KV (memcpy) is cheaper than running a forward pass over p tokens. Measured on Llama-3.2-1B-Instruct, AMD W7900, FlexAttention + torch.compile.

End-to-end throughput (tok/s)

At p=16,384: Cartridge achieves 44.5 tok/s vs ICL at 27.4 tok/s (1.6x end-to-end). The speedup comes entirely from prefill savings; decode throughput is identical.

vLLM TTFT on H100

On H100, prefill at 4K tokens takes only ~80µs of GPU time. TTFT is dominated by Python/HTTP overhead (~5ms), vLLM scheduling (~10ms), and network latency (~3ms). The cartridge advantage manifests at longer prefixes, lower-end GPUs, or higher concurrency.

Quality evaluation

LLM-as-judge: 10 factual questions per document, each answer scored 0–5. Instruct models self-judge; Llama-2-7b answers judged by Qwen-7B.

Qwen2.5-7B-Instruct

Key findings

Instruct tuning is critical. At p=256, instruct models score 2.4–4.0 while the base model scores 0.0–3.2 with high variance.

Model size matters less than expected. Qwen-1.5B (1.5B params) often matches Qwen-7B (7B params), suggesting cartridge training adapts well to smaller models.

Prefix Tuning

Cartridges build on Prefix Tuning (Li and Liang, 2021). The core idea: prepend p trainable continuous vectors to the key and value matrices at every attention layer. These vectors are optimized via gradient descent while the model weights stay frozen.

Initialization strategy matters

Li and Liang found that initialization strongly affects final quality:

First-k initialization (used by cartridges) is the natural extension: the initial KV state comes from actually processing the document, giving the optimizer a warm start in a region of KV space the model already understands. The optimizer then refines these vectors to compress the full document's information into p positions.

From fine-tuning to inference optimization

Original prefix tuning was a fine-tuning technique: train task-specific prefixes to adapt a frozen model. Cartridges repurpose it as an inference optimization: the trained prefix is the document representation, and injecting it replaces prefill entirely.

SCI vs First-k

SCI (Sampled Chunk Initialization) samples random 64-token chunks from across the document instead of taking the first p tokens. In theory, this captures broader content. In practice, both strategies produce nearly identical results:

SCI's broader initialization does not translate into measurable quality or performance advantage over First-k.

vLLM integration

CartridgeConnector plugs into vLLM v0.16.0 as a KVConnectorBase_V1 plugin with zero vLLM core modifications. About 300 lines of Python.

Request lifecycle

Minimum requirements

Server startup

export CARTRIDGE=/path/to/cartridge.pt
export PREFIX_IDS=/path/to/prefix_token_ids.json

vllm serve Qwen/Qwen2.5-7B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.40 \
  --kv-transfer-config '{
    "kv_connector": "CartridgeConnector",
    "kv_connector_module_path": "vllm.distributed.kv_transfer.kv_connector.v1.cartridge_connector",
    "kv_connector_extra_config": {
      "cartridge_path": "'"$CARTRIDGE"'",
      "prefix_token_ids_path": "'"$PREFIX_IDS"'"
    },
    "kv_role": "kv_both"
  }'

Connector implementation

class CartridgeConnector(KVConnectorBase_V1):
    def __init__(self, rank, local_rank, config):
        self.cartridge = torch.load(config.cartridge_path, map_location="cpu")
        self.prefix_token_ids = load_json(config.prefix_token_ids_path)
        self.layer_k = [layer["k"].contiguous() for layer in self.cartridge]
        self.layer_v = [layer["v"].contiguous() for layer in self.cartridge]

    def get_num_new_matched_tokens(self, request, **kwargs):
        # Compare request tokens against stored prefix, block-aligned
        matched = longest_common_prefix(
            request.prompt_token_ids, self.prefix_token_ids
        )
        return align_down_to_block_size(matched, block_size=16)

    def start_load_kv(self, connector_meta, **kwargs):
        # Scatter pre-trained KV from CPU to vLLM-assigned GPU slots
        slot_mapping = connector_meta["slot_mapping"]
        matched = connector_meta["matched_tokens"]
        for layer_idx in range(len(self.layer_k)):
            scatter_to_gpu_blocks(
                self.layer_k[layer_idx][:matched],
                self.layer_v[layer_idx][:matched],
                slot_mapping[:matched],
            )

Sending requests

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-7B-Instruct",
    "messages": [
      {"role": "system",
       "content": "<full training document text>"},
      {"role": "user",
       "content": "What is the main contribution?"}
    ]
  }'

Cross-model analysis

Key findings

GQA models produce dramatically smaller cartridges. 8.5x smaller than MHA at the same prefix. GQA is strongly preferred.

Instruct tuning is critical for compression. Base Llama-2-7b scores 0.0–3.2 at 4x+ compression vs 2.4–4.1 for instruct models.

Small instruct models are surprisingly effective. Qwen-1.5B (1.5B params) often matches or exceeds Qwen-7B (7B params) at compressed cartridge sizes.

Concurrency scaling (H100)