This page asks a narrow question and answers it honestly: of the knlp ideas that touch training and the KV cache, which are worth contributing to Marin, Stanford’s fully-open LLM project? It first lays out what Marin actually welcomes, then scores each candidate we have evidence for against both Marin’s architecture and Marin’s own metric. This is a fit-assessment map, which may be useful for future R&D evaluation.
Marin is an “open lab”: a fully transparent foundation-model effort built on Levanter (JAX, on TPU), where the code, data, checkpoints, and training logs are all public. It runs research the way a journal runs review. Every experiment begins as a GitHub issue that acts as a small preregistration — it states the hypothesis, the regime (for example, “1.4B-parameter models for 28B tokens”), and the metric — and is then submitted as a pull request that declares the experiment in code as a DAG of executor steps. The PR is reviewed openly, “in the spirit of OpenReview,” carries a small-scale sanity check plus a dry run so reviewers can see the cost before it is launched, and the results land back in the issue.
They are explicit that outside method work is in scope: “You are encouraged to try out new architectures, optimizers, and even data filtering strategies.” The lightweight entry path is the Marin Speedrun — pick a compute budget and build the fastest method to train a model to a target quality. A submission is a training config plus a results file, sent as a PR and ranked on the leaderboard by Paloma C4-EN bits-per-byte at a FLOPs budget. (Paloma is AI2’s perplexity benchmark across many text domains; the C4-EN split is the English C4 web-text corpus; and bits-per-byte is the model’s cross-entropy normalized by UTF-8 bytes rather than tokens, which keeps it comparable across tokenizers — lower is better.) The tracks are deliberately small and laddered — 30M, 75M, 300M, 1.4B — and they ask that a method be judged on a scaling suite: “assess a method based on how well it scales (what is the slope?) rather than how good it is at any one scale.” Full experiment runs are scored with their fork of the lm-evaluation-harness (the CORE_TASKS set: ARC, HellaSwag, MMLU, and friends).
The thing to hold onto: Marin’s yardstick is training quality per unit of compute — bits-per-byte at a FLOPs budget, and the slope of that curve across scales. Their own wishlist points the same way (efficient linear attention, long context). A contribution lands when it moves that curve. That single fact decides most of what follows.
Three knlp ideas are close enough to Marin’s training stack to be worth weighing. Each is scored on two independent gates — can it even be expressed on Marin’s architecture, and does it move Marin’s metric — followed by what we actually have.
| Candidate | Ports to Marin’s architecture? | Moves Marin’s metric (BPB/FLOP)? | Verdict |
|---|---|---|---|
| FIM-guided pruning | Yes — an Optax GradientTransformation over Adam’s second moment; drops into Levanter | No — pruning during training doesn’t cut training FLOPs or bend the loss-vs-FLOP curve | Only clean port, but off-axis |
| KVSplice | No — its edge needs MLA; Marin is GQA (8B Llama, 32B Qwen3-style) | No — an inference-memory lever; KV-cache bytes aren’t on the leaderboard | Architecturally blocked |
| Reciprocal attention | In principle — a structural attention change | Unproven — the small-scale signal did not survive scale-up | Not ready — still R&D |
This is the one candidate that fits Marin’s stack without contortion. Pruning is scored by |W| × exp_avg_sq^0.25 — Adam’s own second moment, which Marin already carries in Optax state — so it can be written as a custom GradientTransformation and dropped into a Levanter training run. We have a clean from-scratch result at a Marin llama_150m config: FIM-guided pruning beats magnitude pruning at every sparsity level and beats the dense baseline at 25% (7.69 vs 7.73 PPL).
The honest catch is the second gate. Marin ranks methods by quality at a training-compute budget, and pruning during training reduces neither the training FLOPs (the gradients are still dense) nor the loss-vs-FLOP frontier. It makes the model sparser at a given quality — a real inference and deployment win, but one Marin’s leaderboard does not measure. Marin would entertain it as a one-off preregistered experiment; it is not something they are asking for, and it will not climb the Speedrun.
KVSplice’s whole value proposition is extra compression stacked on the already-latent cache of Multi-head Latent Attention — the 7.2× = 6× MLA × 1.2× splice story. Marin uses no MLA. Marin-8B is plain Llama with grouped-query attention (32 query / 8 KV heads, already a 4× KV reduction); Marin-32B is Qwen3-style (Llama plus QK-norm), also GQA. Strip MLA away and KVSplice is just a learned low-rank KV projection competing with the GQA Marin already ships — which is exactly what our own no-MLA 150M runs showed 1.2–2.0× KV-cache compression for a +1.6 to +2.7% PPL cost — modest, and no better than the reduction GQA already gives. (Those compression ratios are the measured KV-cache gain; the KVSplice page carries the per-layer numbers and the byte-level stack.)
And it fails the second gate independently: KVSplice is an inference/decode memory lever, while Marin scores training quality per FLOP. Wrong architecture and wrong axis. There is no clean way to contribute it here.
RA is a structural attention change and is, in principle, the kind of thing Marin would review. But it is still R&D, and its scaling record argues against contributing it. We scaled several RA levers — layer and head selectors (FIM-trace, Fisher eigenmax), head count, and the residual mix — and the small-scale signal did not survive: it tops out around 150M and is neutral-to-negative by 1B. Until there is a new inductive-bias hypothesis and a quality-per-FLOP gain to show, there is nothing here that would move Marin’s curve. The full scaling story is on the RA page.
Only FIM-guided pruning is worth carrying further, and only if it can be made to move Marin’s metric. The work to get there is mostly unstarted — this section is deliberately a checklist of open items, not results.
A Marin contribution is not an 8B-from-scratch run — Marin-8B is 12.7T tokens. It is the Speedrun ladder. Per single training run, token-matched at Chinchilla 20 tokens/param (Marin’s own 1.4B@28B ablation regime):
| Scale | Tokens (20×) | FLOPs | W7900 (1×) | H100 (1×) |
|---|---|---|---|---|
| 30M | 0.6B | 1.1e17 | ~1.2 h | ~5 min |
| 75M | 1.5B | 6.8e17 | ~8 h | ~28 min |
| 300M | 6B | 1.1e19 | ~5 days | ~7.6 h |
| 1.4B | 28B | 2.4e20 | ~110 days* | ~6.9 days* |
Per single run; an A/B needs ×2 (FIM + magnitude). C ≈ 6·N·D; H100 SXM 989.5 TFLOPS BF16 @ 40% MFU; W7900 122.6 TFLOPS BF16 @ ~20% MFU (ROCm/RDNA3). * 1.4B is impractical on one GPU — on an 8×H100 node ~21 h/arm. The 30M–75M rungs run locally on the single W7900 in a day; 300M is a multi-day W7900 grind or a cheap H100 afternoon. The real gate, though, is item 03, not the GPU hours.
By Marin’s own yardstick — quality per training FLOP — none of these is a slam-dunk. KVSplice is architecturally out (Marin isn’t MLA), reciprocal attention is unproven at scale, and FIM-guided pruning, the only one that ports cleanly, is off the axis Marin actually measures. The candidate exists; the case that it would move Marin’s curve does not yet. This page records that honestly rather than dressing a weak fit as a contribution.