Linear attention and bounded memory
Standard causal attention keeps past keys and values in a KV cache, so the stored history grows with context length. The models studied here replace that growing history with a fixed-size recurrent matrix state. Each token modifies the state and later tokens query it.
Across the matched experiments reported here, Gated DeltaNet is the strongest bounded-state baseline. Trellis beats the matched dense Transformer on long-document experiments, but does not beat Gated DeltaNet at the scales tested. Trellis results published at 125M parameters and above were not reproduced here.
Growing KV cache versus fixed recurrent state
In causal attention, a token produces a key and a value. Future queries may attend to them, so inference retains those vectors. If the context doubles, the number of stored key/value vectors also doubles.
A bounded-memory model instead compresses prior tokens into a state whose dimensions are fixed by the architecture. The state can change indefinitely without becoming larger.
Retain one key/value record for every cached token.
Fold token history into a fixed matrix and query that matrix.
The trade-off is compression. Dense attention retains token-level history explicitly. A bounded state must decide what to preserve, overwrite, or forget.
DeltaNet stores key-value associations in a matrix
Let S ∈ Rdᵥ×dₖ be the recurrent memory. For one token:
- k ∈ Rdₖ is the key used to address memory.
- v ∈ Rdᵥ is the value that key should retrieve.
- β is a learned write strength.
- q ∈ Rdₖ is the query used to read memory.
o = Sq
Sk is the value currently associated with k. The error v − Sk measures how far that association is from the desired value. The outer product with kᵀ writes a correction back into the state.
When the current token updates the state before its query reads it, the implementation is write-before-read.
The update can also be written as:
This form exposes the identity-minus-rank-one transition that connects DeltaNet to the WY/UT blocking machinery.
Four ways to update a bounded state
Uses the delta-rule update above. After k, v, and β are known, the old state appears only linearly.
Adds a learned retention factor α so old state can decay:
The update remains affine in S.
Uses several structured Householder-like updates per token. This increases overwrite capacity while keeping the transition affine in the recurrent state.
Uses a learned nonlinear compression objective inside the memory update. Let M ∈ Rm×d be memory, k ∈ Rd an input key, and a ∈ Rm the target memory code:
With z = Mk and u = Jφ(z)ᵀ(φ(z) − a), one write has the form:
φ is the compression nonlinearity, Jφ its Jacobian, β the retention gate, and γ the inner step size. Because u depends nonlinearly on M, this update is not affine in the state.
Affine in the state means that, after token-derived coefficients are fixed, the old state is only multiplied, scaled, and added to other terms. No nonlinear function of the current state appears in those coefficients.
Terms used in the results
| Term | Meaning |
|---|---|
| C4 | Colossal Clean Crawled Corpus, a large cleaned web-text dataset derived from Common Crawl and widely used for language-model training. |
| PG19 | A long-document language-modeling dataset built from Project Gutenberg books. It is useful here because documents are much longer than ordinary short web samples. |
| FLA | Flash Linear Attention, the library supplying the reference DeltaNet, Gated DeltaNet, and GatedDeltaProduct implementations used in the matched comparisons. |
| PPL | Perplexity. Lower is better. It is the exponential of the mean token negative log-likelihood. |
| NLL | Negative log-likelihood. Here it is reported in natural-log units, or nats, per token. Lower is better. |
Gated DeltaNet leads the first-order Trellis comparison
The A100 anchor used C4, model width 256, four layers, sequence length 2048, about 352M training tokens, fp32 master weights with bf16 autocast, FLA reference linear-attention layers, and the fused Trellis Triton kernel.
The Trellis arm used a chunk-start-stale forward: within each chunk, the nonlinear inner gradient is evaluated from the state at the start of the chunk rather than from the running token-by-token state. Its outer gradient was first-order, meaning it did not differentiate through the inner gradient computation. The FLA affine baselines use their exact chunked recurrences.
| Model | Val PPL ↓ | Approx. non-embedding params | Training semantics |
|---|---|---|---|
| Gated DeltaNet | 78.49 | 4.48M | exact affine recurrence |
| DeltaNet | 89.77 | 4.21M | exact affine recurrence |
| Trellis | 152.54 | 4.74M | chunk-start-stale forward, first-order outer gradient |
| dense Transformer | 224.89 | 4.72M | dense causal attention |
This run establishes the ordering for those implementations and semantics. It does not compare Gated DeltaNet against full-bilevel Trellis at a 352M-token budget.
The forward approximation and the outer gradient are independent choices
Chunk-start forward
The Trellis paper divides the sequence into chunks and evaluates all nonlinear inner gradients in a chunk using the state at the beginning of that chunk. The paper explicitly presents this as an approximation that locally linearizes the nonlinear recurrence and enables a parallel scan.
The exact running-state recurrence instead recomputes each token's nonlinear write from the state produced by all earlier tokens. That path is sequential inside the chunk.
First-order versus full bilevel outer gradient
Trellis writes memory by taking a gradient of an inner objective. Training the surrounding language model can either differentiate through that inner gradient or treat the inner correction as a fixed value.
Detach the state used to form the inner gradient. The forward memory update is unchanged, but the higher-order derivative path is removed.
Differentiate through the inner gradient. This introduces second-derivative terms, commonly evaluated as Hessian-vector products.
The two modes can therefore produce identical forward values and different parameter gradients.
Gated DeltaNet and identity-bilevel Trellis tie at 20M tokens; Gated DeltaNet leads at 40M
These runs use model width 256, four layers, sequence length 2048, three paired seeds, the same packed C4 stream, and explicit Trellis forward/gradient semantics. For the Trellis write comparison, the inter-pass activation is held fixed while the compression write changes.
Write nonlinearity depends on gradient semantics
| Trellis write | Outer gradient | 20M result | Stability |
|---|---|---|---|
| identity / linear | first-order | ~253.2 PPL across clean completions | 3 clean completions / 4 attempted seeds |
| SiLU nonlinear | first-order | 251.6 PPL | 3/3 |
| identity / linear | full bilevel | 244.5 PPL | 3/3 |
| SiLU nonlinear | full bilevel | no stable three-seed mean | 1/3 completed healthily |
Under first-order training, SiLU is slightly better than the identity write. Under full-bilevel training, the identity write improves substantially while full-strength SiLU becomes unstable. The write cannot be ranked without stating the outer-gradient semantics.
Budget extension
| Arm | 20M mean NLL ↓ | 40M mean NLL ↓ | 40M PPL ↓ |
|---|---|---|---|
| Gated DeltaNet | 5.4965 | 5.1033 | 164.6 |
| identity-bilevel Trellis | 5.4981 | 5.1132 | 166.2 |
| SiLU first-order Trellis | 5.5300 | 5.1596 | 174.1 |
At 20M tokens, Gated DeltaNet and identity-bilevel Trellis differ by only 0.0016 nat/token on mean NLL. At 40M, the Gated DeltaNet lead grows to 0.0099 nat/token. One seed still slightly favors identity-bilevel, but all three paired seed gaps move toward Gated DeltaNet as the budget increases.
Tempering the higher-order gradient
A tempered Trellis variant scales only the higher-order gradient path:
The forward value of u is unchanged. With ρ = 0.5, all three seeds completed, where full-strength ρ = 1 SiLU lost two of three seeds. The tempered arm beat its paired first-order SiLU run in all three seeds by about 0.006 nat/token on average. The gain is small; the main result is the stability recovery.
PG19 shows a long-context advantage over the matched dense baseline
The PG19 comparison uses three seeds and held-out perplexity. These runs predate the later bilevel study: Trellis uses a first-order outer gradient. The operator column states whether the forward uses the exact running state or the chunk-start approximation.
| Context | Trellis forward | Trellis PPL | Dense PPL | Read |
|---|---|---|---|---|
| 512 | running-state | 1542 ± 31 | 1581 ± 131 | tie; dense variance is large |
| 1024 | chunk-start, C=16 | 1359 ± 6 | 1665 ± 114 | Trellis lower |
| 1024 | running-state confirmation | 1517 ± 91 | 1635 ± 69 | win holds, high variance |
| 2048 | chunk-start, C=16 | 534 ± 31 | 616 ± 11 | clean non-overlap |
| 2048 | running-state confirmation | 585 ± 9 | 621 ± 11 | win holds |
| 4096 | chunk-start, C=16 | 546 ± 1.5 | 622 ± 5 | clean non-overlap |
The 512-token cell is not evidence for a short-context win. The clear signal appears at longer contexts, especially 2048 and 4096. Running-state confirmations at 1024 and 2048 show that the dense-beating direction is not created solely by the chunk-start approximation.
On the same PG19 harness, Gated DeltaNet remains substantially stronger than Trellis: at context 1024, Gated DeltaNet reaches 1188 PPL versus Trellis 1517 and dense 1635; at 2048, it reaches 423 versus Trellis 585 and dense 621.
DeltaNet composes exact state maps; Trellis linearizes its nonlinear write
For a token whose state update has the form
with A and B fixed after the token quantities are known, the update is affine in S. Composing two such updates gives another affine update, and the same remains true for an entire chunk.
DeltaNet and Gated DeltaNet have this property. Their rank-one transitions can be collected into compact factors and a triangular dependency solve, then applied with matrix multiplications. The historical route from Householder reflectors to this WY/UT representation is described separately.
Trellis does not satisfy the same condition because u = Jφ(Mk)ᵀ(φ(Mk) − a) changes nonlinearly with the running memory M. The paper therefore evaluates those nonlinear gradients from the state at the beginning of each chunk, making the within-chunk recurrence locally linear and parallelizable by scan.
The fused Trellis kernel removes launch overhead but remains slower than the affine kernels
Mixer-level forward-plus-backward throughput on an NVIDIA A100 with batch 8, sequence length 2048, model width 256, and four heads:
| Mixer | Tokens/s | Interpretation |
|---|---|---|
| FLA Gated DeltaNet | 3.891M | fastest measured |
| FLA DeltaNet | 3.315M | faster than dense attention in this mixer benchmark |
| dense multi-head attention | 2.941M | dense reference |
| Trellis fused Triton | 0.770M | about 4× below Gated DeltaNet |
| Trellis Python | 0.121M | launch-bound implementation |
The fused kernel raises Trellis from about 0.121M to 0.770M tokens/s, a 6.4× gain. The remaining gap is a measurement of these implementations, not a lower bound on nonlinear recurrent memory.
Decode has a different trade-off. A bounded state does not grow with context, but it is still a dense read-modify-write object every generated token. Its crossover against a grouped-query-attention KV cache depends on state size, precision, batch, context length, and hardware.
Extra overwrite machinery does not improve the 20M-token C4 pilot
GatedDeltaProduct-2 uses two structured state micro-steps per token. On the synthetic overwrite/binding test it reaches 1.000 while Gated DeltaNet reaches 0.996, so both solve the intended overwrite task.
The language-model pilot uses C4, model width 512, ten layers, sequence length 2048, 20M tokens, and three seeds. It is width-matched rather than parameter-matched: GatedDeltaProduct-2 has about 49.9M parameters versus 44.7M for Gated DeltaNet and performs more recurrent work per token.
| Model | Val PPL ↓ | Notes |
|---|---|---|
| Gated DeltaNet | 215.72 | 44.7M params |
| GatedDeltaProduct-2 | 224.39 | 49.9M params; two micro-steps/token |
| DeltaNet | 254.50 | FLA reference |
| dense Transformer | 377.91 | dense reference |
Gated DeltaNet leads GatedDeltaProduct-2 by about 0.039 nat/token in this early-training pilot. This result is specific to this scale and training budget.
The paper reports Trellis wins at larger scales
The Trellis paper reports results well above the model and token budgets used in this study.
| Published setting | Trellis | Gated DeltaNet | Result |
|---|---|---|---|
| 125M / 2.4B tokens, The Pile, 2K context | 10.87 PPL | 11.31 PPL | Trellis lower |
| large C4 LM, 30B tokens | 20.28 PPL | 21.40 PPL | Trellis lower |
| 1B, RULER S-NIAH | 79.8 average | 75.8 average | Trellis higher |
RULER is a synthetic long-context evaluation suite. S-NIAH denotes its single-needle-in-a-haystack retrieval tasks.
There is a small parameter-count inconsistency inside the paper: Table 1 labels the large language-model comparison 790M parameters / 30B tokens, while the appendix architecture table lists the corresponding large model as 780M / 30B. The values above are the Table 1 language-model results.
The 125M ablation also reports a linear compression write with the inter-pass activation held at LN-SiLU: 11.65 PPL versus 10.87 for full Trellis. At that published scale, the nonlinear write improves language-model perplexity by 0.78.
None of the 125M-and-larger published scale claims were reproduced in this study.
Gated DeltaNet is the practical baseline for the measured regime
The experiments support four separate conclusions:
- Bounded recurrent memory can beat the matched dense Transformer on long-document language modeling; the clean PG19 signal appears at longer contexts, not at 512 tokens.
- Within the bounded-state family tested here, Gated DeltaNet has the strongest language-model results and the fastest measured training mixer.
- Trellis's nonlinear write changes both chunking and training-gradient behavior. Full-bilevel gradients help the linear identity write at 20M tokens but destabilize full-strength SiLU; tempering the higher-order path restores stability with a small gain.
- The published Trellis advantage at 125M parameters and above remains an untested scaling result, not a result contradicted by these smaller experiments.