Follow-on experiments built from the Qwen3-8B LongHealth reproduction. These studies change the initialization, objective, or serving format and are separate from reproduction of the Cartridges at Scale paper.
CAS reproduction · asymmetric cartridge quantization · research harness
The main CAS reproduction page reports the paper recipe, LongHealth evaluation, cache-path controls, implementation details, and observations made while running that recipe. This page collects experiments that begin from the reproduced cartridge and then change the method.
| Work | Question | Result |
|---|---|---|
| Meta-initialization | Can other trained cartridges shorten a new run? | No useful speedup |
| Reverse KL | Can a different objective improve free generation? | No resolved gain |
| Asymmetric quantization | Can a cartridge use FP8 without losing key fidelity? | 0.029%–0.038% loss increase with K16/V8 |
This experiment fits a shared early-training displacement from other documents and applies it to a new cartridge before training. On patient 02, the best step-60 affine initialization lowered step-zero held-out loss, but the ordinary training trajectory quickly absorbed the difference.
| Measurement | Result |
|---|---|
| Head start at 50% of the baseline loss drop | 3.3 optimizer steps |
| Head start at 75% of the baseline loss drop | 5.0 optimizer steps |
| Head start at 90% of the baseline loss drop | 5.6 optimizer steps |
| Difference by step 180 | less than 0.00003 loss |
| Same-seed repeat through step 110 | bit-for-bit identical |
The faithful run takes roughly 1,000 optimizer steps. Saving at most six steps, with no persistent loss difference after step 180, does not provide a useful training-speed improvement at this scale.
The pilot warm-started patient 02 from the 0.65 cartridge, optimized a reverse Kullback–Leibler objective for 200 steps at a learning rate of 0.005, and evaluated the resulting cartridge with three Table-15 generation runs.
| Cartridge | Accuracy | Correct answers |
|---|---|---|
| Faithful CAS starting point | 0.65 ± 0.05 | 39 / 60 |
| Reverse-KL step 200 | 0.70 ± 0.05 | 42 / 60 |
The three-answer difference is smaller than one binomial standard deviation. This evaluation does not resolve an accuracy improvement.
The quantization study treats a trained cartridge as a deployment artifact rather than changing how it is trained. Across three separately trained Qwen3-8B cartridges, FP8 values with BF16 keys increased the cartridge objective by 0.029%–0.038%. Quantizing both keys and values increased it by 0.873%–1.303%.
The serving implementation keeps one 632-position K16/V8 cartridge shared across requests and attends over it together with each request's live cache in one FlashInfer kernel. On one H100 SXM 80 GB, the shared path improved both throughput and time to first token against ordinary BF16 cartridge injection at batches 1, 16, and 64.
The public harness contains the checkpoint reader, held-out cartridge-loss evaluator, meta-initialization fitter, and paired learning-curve comparison. The serving work is implemented in the FlashInfer and vLLM branches linked from the asymmetric cartridge quantization page.