Cartridges

CartridgeConnector Serving Validation

Full validation of the 20260429-cartridges-code-only branch. Validates cartridge KV injection, per-request dispatch, manifest validation, GPU residency, and serving correctness.

Tier results

Bugs found & fixed

Unit test coverage

All tests run CPU-only via pytest tests/v1/core/test_cartridge*.py. No GPU required.

How to reproduce

# Clone the branch
git clone --branch 20260429-cartridges-code-only https://github.com/mcgrof/vllm.git
cd vllm

# Unit tests (no GPU needed)
uv venv --python 3.12 && source .venv/bin/activate
VLLM_USE_PRECOMPILED=1 uv pip install -e . --torch-backend=auto
pytest tests/v1/core/test_cartridge*.py -v

# Smoke serve (needs GPU + trained cartridge .pt)
vllm serve meta-llama/Llama-3.2-3B-Instruct \
  --kv-transfer-config '{
    "kv_connector": "CartridgeConnector",
    "kv_connector_module_path":
      "vllm.distributed.kv_transfer.kv_connector.v1.cartridge_connector",
    "kv_connector_extra_config": {
      "cartridge_path": "/path/to/cartridge.pt"
    },
    "kv_role": "kv_both"
  }'

# Multi-cartridge dispatch
vllm serve meta-llama/Llama-3.2-3B-Instruct \
  --kv-transfer-config '{
    "kv_connector": "CartridgeConnector",
    "kv_connector_module_path":
      "vllm.distributed.kv_transfer.kv_connector.v1.cartridge_connector",
    "kv_connector_extra_config": {
      "cartridges": [
        {"cartridge_id": "doc_a", "path": "/path/to/cart_a.pt"},
        {"cartridge_id": "doc_b", "path": "/path/to/cart_b.pt"}
      ],
      "router": {"type": "explicit"}
    },
    "kv_role": "kv_both"
  }'

Reproducing the Tests

Full guide: cartridges_reproduce.md

Automated (knlp pipeline)

# Clone knlp, select cartridges config, run everything
git clone https://github.com/mcgrof/knlp.git && cd knlp
make defconfig-cartridges-vllm-tests
make

# Pipeline: doctor → fetch → build → test → report
# Reuses existing vllm checkout if present (no re-clone)
# Runs 178 unit tests (no GPU needed for Tier -1)

# Run a single tier
make cartridges-test TIER=-1   # unit tests only
make cartridges-test TIER=6    # TP=2 only (needs 2 GPUs)

Manual: unit tests (no GPU)

git clone --branch 20260429-cartridges-code-only \
  https://github.com/mcgrof/vllm.git && cd vllm
VLLM_USE_PRECOMPILED=1 uv venv --python 3.12
source .venv/bin/activate
VLLM_USE_PRECOMPILED=1 uv pip install -e . --torch-backend=auto
uv pip install transformers==4.55.4 pytest pytest-timeout tblib

pytest tests/v1/core/test_cartridge*.py -v --timeout=120
# Expected: 178 passed

Manual: serve with cartridge (needs GPU)

# Single cartridge
vllm serve meta-llama/Llama-3.2-3B-Instruct \
  --kv-transfer-config '{
    "kv_connector": "CartridgeConnector",
    "kv_connector_module_path":
      "vllm.distributed.kv_transfer.kv_connector.v1.cartridge_connector",
    "kv_connector_extra_config": {
      "cartridge_path": "/path/to/cartridge.pt"
    },
    "kv_role": "kv_both"
  }'

# Multi-cartridge with per-request dispatch
vllm serve meta-llama/Llama-3.2-3B-Instruct \
  --kv-transfer-config '{
    "kv_connector": "CartridgeConnector",
    "kv_connector_module_path":
      "vllm.distributed.kv_transfer.kv_connector.v1.cartridge_connector",
    "kv_connector_extra_config": {
      "cartridges": [
        {"cartridge_id": "doc_a", "path": "cart_a.pt"},
        {"cartridge_id": "doc_b", "path": "cart_b.pt"}
      ],
      "router": {"type": "explicit"}
    },
    "kv_role": "kv_both"
  }'

# TP=2 (needs 2 GPUs)
vllm serve meta-llama/Llama-3.2-3B-Instruct \
  --tensor-parallel-size 2 \
  --kv-transfer-config '{...same as above...}'

Building a test cartridge from any text

# No trained cartridge needed — prefill any text to create one
python3 -c "
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, DynamicCache

tok = AutoTokenizer.from_pretrained('meta-llama/Llama-3.2-3B-Instruct')
model = AutoModelForCausalLM.from_pretrained(
    'meta-llama/Llama-3.2-3B-Instruct',
    torch_dtype=torch.bfloat16).to('cuda').eval()

text = 'Your document text here. The model will use this as context.'
ids = tok.encode(text, add_special_tokens=False)
cache = DynamicCache()
with torch.no_grad():
    model(input_ids=torch.tensor([ids], device='cuda'),
          past_key_values=cache, use_cache=True, return_dict=True)

Ks, Vs = [], []
for li in range(len(cache.layers)):
    layer = cache.layers[li]
    k = layer.keys.cpu() if hasattr(layer, 'keys') else layer.key_cache[0].cpu()
    v = layer.values.cpu() if hasattr(layer, 'values') else layer.value_cache[0].cpu()
    Ks.append(k); Vs.append(v)

torch.save({'trainable_keys': Ks, 'trainable_values': Vs,
            'frozen_keys': [], 'frozen_values': []}, 'my_cartridge.pt')
print(f'Saved: {len(Ks)} layers, {Ks[0].shape[-2]} tokens')
"

Key points for AI assistants reproducing these tests

• Branch: 20260429-cartridges-code-only on mcgrof/vllm
• The tokenizer needs a one-line patch for all_special_tokens_extended (fork-point compat — automated in the knlp pipeline)
• Use VLLM_USE_PRECOMPILED=1 to avoid compiling C extensions from source
• Prompts must include the document text as prefix for the cartridge context to work
• The CartridgeConnector replaces KV for the first N tokens of the prompt — it does not add tokens
• TP>1 works: the connector shards KV heads across ranks automatically
• Unit tests are CPU-only and cover connector, manifest, registry, store, router, dispatch, residency, LMCache plugin, and fault paths