LayerRoPE and mHC.
What the experiments showed.
We tested whether changing a model’s residual connections could improve learning and make its key/value (KV) cache easier to compress. On these small language models, learning more from each token did not mean learning more in the same time. Ordinary residual connections finished with the best predictions when every model had the same time budget.
What we learned
The simple baseline finished with the lowest loss, before and after K3/V2 cache rounding. It completed far more updates. LayerRoPE came second; adding mHC did not earn its runtime cost here. See the eight-seed comparison →
With every model limited to 2,000 updates, mHC + LayerRoPE had the lowest mean loss. That earlier result showed better use of training data, but each update took much longer. See the original comparison →
Each architecture’s smallest tested cache stayed at 3,072 theoretical bytes per token when rounding could add at most 0.1 nat of loss. QKNorm did not change that result. See the two quality tests →
The most promising follow-up was QKNorm. Normalizing attention’s queries and keys improved mHC in all three matched runs at equal time. It still did not catch the residual baseline. A smaller-cache result appeared at equal updates under a shared quality target; it disappeared when training time was held equal. The QKNorm results explain why.
These are 32-layer, roughly 20-million-parameter TinyStories models with 256-token contexts. The conclusions describe this implementation and training budget. Cache sizes include scale metadata but are calculated: the experiments simulated low-precision rounding, without packing a smaller serving cache.
How to read the results
Loss measures how well the model predicts the next token; lower is better. Tables use cross-entropy (CE), in nats. Clean means evaluation without the extra quantization being tested. Perplexity is exp(CE).
K3/V2 means three bits for cached keys and two for cached values. Comparing it with K3/V3 measures the extra loss from removing one value bit. A model can lose less from that change and still finish worse than a stronger model.
A seed sets the random initialization and training order. Paired runs share those choices across models. The equal-token, equal-time and QKNorm experiments use separate seeds and evaluation sets, so compare models within each experiment rather than treating their losses as one continuous training curve.
The combination learned more from the same training data
Tables scroll horizontally on narrow screens.
We began with a simple comparison: give every model the same 4.096 million training tokens and see which learns most. We tested ordinary residual connections, mHC, and three initial settings for LayerRoPE, both alone and combined with mHC. Each of the eight configurations had the same learning-rate search budget and three matched training runs.
The combination with LayerRoPE’s beta parameters initialized to zero had the lowest mean prediction loss. Beta controls how its gains vary with layer depth; zero is a starting setting, not a switch that turns LayerRoPE off. The interactive explanation shows the difference.
| Model | Selected LR | Parameters (M) | Average validation CE | Final clean CE | PPL | Training seconds |
|---|---|---|---|---|---|---|
| Residual | 0.003 | 19.297 | 2.690108 | 2.765036 | 15.880 | 121.1 |
| mHC | 0.003 | 20.118 | 2.619591 | 2.692526 | 14.769 | 615.0 |
| LayerRoPE, β=−1.0 | 0.01 | 19.289 | 2.709233 | 2.772207 | 15.994 | 231.8 |
| LayerRoPE, β=−0.5 | 0.01 | 19.289 | 2.667277 | 2.729193 | 15.321 | 250.4 |
| LayerRoPE, β=0.0 | 0.01 | 19.289 | 2.651701 | 2.727627 | 15.297 | 254.4 |
| mHC + LayerRoPE, β=−1.0 | 0.003 | 20.110 | 2.766845 | 2.829792 | 16.942 | 826.4 |
| mHC + LayerRoPE, β=−0.5 | 0.01 | 20.110 | 2.654810 | 2.715576 | 15.113 | 786.7 |
| mHC + LayerRoPE, β=0.0 | 0.01 | 20.110 | 2.602404 | 2.676804 | 14.539 | 786.8 |
Adding beta-zero LayerRoPE to mHC reduced average validation loss by 0.017187 nats and final clean loss by 0.015722, with improvement in all three matched runs. That was a real lead in this experiment. The unanswered question was whether it justified the extra training time.
Each configuration selected its own learning rate from the same four candidates. This compares tuned recipes, so the differences cannot be assigned to the architecture alone. Several selected the highest rate tested, 0.01; a wider search might change the result.
Validation curves for all beta initializations
At the same training time, ordinary residual connections won
We then gave each model 787 seconds—about 13 minutes—of training. The residual baseline completed about 11,534 updates, compared with 2,076 for mHC and 1,655 for the combination. That extra training paid off: residual finished with the lowest mean loss, and standalone LayerRoPE was second.
This comparison used eight new matched runs per model, a fresh evaluation set and the same five-candidate learning-rate search budget for every architecture. It did not resume the earlier models. Both LayerRoPE beta parameters started at zero and remained trainable. In the table, clean means no added rounding; K3/V2 uses three-bit keys and two-bit values.
| Model | Clean CE | K3/V3 CE | K3/V2 CE | K3/V2 − K3/V3 | Mean updates |
|---|---|---|---|---|---|
| Residual | 2.354792 | 2.380044 | 2.442863 | +0.062819 | 11,534.000 |
| LayerRoPE, β=0 | 2.483780 | 2.498185 | 2.533388 | +0.035203 | 5,438.125 |
| mHC | 2.771197 | 2.783344 | 2.813972 | +0.030627 | 2,076.250 |
| mHC + LayerRoPE, β=0 | 2.828159 | 2.837663 | 2.866000 | +0.028338 | 1,655.125 |
Residual and LayerRoPE each beat both mHC versions in all eight matched runs, both before rounding and with a K3/V2 cache. All eight corresponding comparisons also passed the preselected 0.01-nat improvement rule for the median paired difference. Residual had the lowest observed mean; a formal residual-versus-LayerRoPE claim was outside that test family. The protocol gives the statistical details.
mHC lost less from the extra value-bit rounding, but still produced worse predictions. Its K3/V2 loss was 2.813972, versus 2.442863 for residual. This is why a small quantization penalty alone is not enough: what matters is the quality of the model you end up using.
All four architectures supported the same smallest tested cache: 3,072 theoretical bytes per token, including metadata, with no more than 0.1 nat of added loss in every run. None supported the smaller 2,560-byte format across all eight runs.
What counted as training time?
The clock included preparing and transferring batches, warm-up, forward/backward computation, optimizer updates and scheduled numerical checks. It excluded setup, offline evaluation and checkpoint writes. Each checkpoint followed a complete update, with at most one update and 1% overshoot. This measures training efficiency; total experiment cost also includes the excluded work.
The selected learning rates were 0.001 for residual, 0.01 for LayerRoPE, 0.003 for mHC and 0.01 for the combination. These timings come from the portable implementation used here. Optimized kernels could change the tradeoff.
The mHC runs also exceeded our chosen tolerance for how accurately their routing weights should be normalized. Their values remained finite, and every run stays in the comparison. A separate numerical investigation found no qualified fix; it did not show that routing error caused the learning gap.
QKNorm helped mHC, but did not close the gap
Could the attention calculation itself help? We added Query-Key Normalization (QKNorm) to each architecture and repeated the comparison with three new matched runs, training 24 models in total. QKNorm reduced mHC’s mean clean loss from 2.722623 to 2.653771 at the same time budget, improving all three runs. Residual without QKNorm still led, at 2.280066.
QKNorm normalizes each query and key vector to unit length, then learns a scale for the attention scores. It adds no normalization to values. LayerRoPE instead controls how layers read and write their state; mHC mixes several residual streams. The changes can therefore be tested together.
We kept each architecture’s previous learning rate when switching QKNorm on. This was a focused test of that change, not a new search for the best settings. Its three runs are separate from the eight-run equal-time experiment above.
The benefit depends on what you hold fixed
Switch between 787 seconds of training and 1,000 updates. Each update processes 2,048 training tokens. Both checkpoints are scored on the same 131,072 held-out targets. Lower bars mean better predictions.
787 training seconds · mean clean CE · QKNorm off versus on
Residual
On minus off: +0.011829 nats
LayerRoPE
On minus off: −0.008396 nats
mHC
On minus off: −0.068852 nats
mHC + LayerRoPE
On minus off: −0.001448 nats
All bars use the same zero-based 0–3.2 nat scale. The full table below remains available without JavaScript. Means and paired ranges describe these three seeds; ranges are not confidence intervals.
| Endpoint | Architecture | Off clean CE | On clean CE | Mean delta | Paired delta range |
|---|---|---|---|---|---|
| 787 seconds | Residual | 2.280066 | 2.291895 | +0.011829 | +0.001674 to +0.018557 |
| 787 seconds | LayerRoPE | 2.441022 | 2.432625 | −0.008396 | −0.018279 to +0.004146 |
| 787 seconds | mHC | 2.722623 | 2.653771 | −0.068852 | −0.085912 to −0.057450 |
| 787 seconds | mHC + LayerRoPE | 2.766167 | 2.764719 | −0.001448 | −0.023085 to +0.012058 |
| 1,000 updates | Residual | 3.039782 | 2.991183 | −0.048599 | −0.059928 to −0.036639 |
| 1,000 updates | LayerRoPE | 2.960515 | 2.938223 | −0.022292 | −0.033617 to −0.005758 |
| 1,000 updates | mHC | 2.938937 | 2.886090 | −0.052847 | −0.069344 to −0.021662 |
| 1,000 updates | mHC + LayerRoPE | 2.917502 | 2.912387 | −0.005116 | −0.039031 to +0.024791 |
At 1,000 updates, QKNorm improved residual, LayerRoPE and mHC in every matched run; mHC with QKNorm had the lowest mean loss. But QKNorm also made the residual baseline slower. Within 787 seconds it completed 8,276–8,922 updates, versus 9,999–11,371 without QKNorm, and finished worse in all three pairs. Better updates did not compensate for fewer updates. LayerRoPE and the combination showed mixed results at equal time.
Did QKNorm buy a smaller cache?
At equal training time, no. The answer at equal updates needs more care, because we tested two different quality requirements:
How much did rounding hurt this model?
Compare the quantized model with its own unrounded version. Allow at most 0.1 nat of added loss. Under this test, every architecture still needed at least 3,072 theoretical bytes per cached token, with QKNorm off or on, at both checkpoints.
Is the final model good enough?
Compare every model with the same unrounded residual baseline, allowing 0.1 nat above its loss. A better starting model can spend some of that quality advantage on stronger rounding. This tests the final result, including both training quality and quantization damage.
The table uses the second test: one shared quality target, set by the paired residual model without QKNorm at the same checkpoint. One fixed cache format must pass every seed. “None” means none of the tested formats met that target, even before rounding.
| Architecture | 787 seconds | 1,000 updates |
|---|---|---|
| Residual | 3,072 → 3,072 | 3,072 → 3,072 |
| LayerRoPE | None → None | 2,560 → 2,560 |
| mHC | None → None | 3,072 → 2,560 |
| mHC + LayerRoPE | None → None | 2,560 → 2,560 |
At 1,000 updates, QKNorm let mHC use signed K2/V2 and still meet the shared target: 2,560 instead of 3,072 theoretical bytes per token, a 16.7% reduction. Its better starting quality helped make that possible. The tolerated damage relative to its own unrounded version did not improve enough to change that model-relative minimum. LayerRoPE and the combination already passed at 2,560 bytes without QKNorm.
At equal time, only residual met the shared target, at 3,072 bytes with or without QKNorm. The other architectures already missed it before cache rounding. The equal-update saving therefore did not become an equal-time advantage.
K3/V3 versus K3/V2: inspect both endpoints
K3/V3 is the reference for lowering V alone from three to two bits while holding K at three. Both use the same signed codebook rule; unequal K/V widths are a separate choice from signed versus affine quantization. K3/V3 costs 3,584 ideal bytes per token and K3/V2 costs 3,072.
| Endpoint | Architecture | QKNorm | K3/V3 CE | K3/V2 CE | Added CE: V3 → V2 |
|---|---|---|---|---|---|
| 787 seconds | Residual | off | 2.307041 | 2.371075 | 0.064035 |
| 787 seconds | Residual | on | 2.314419 | 2.368250 | 0.053832 |
| 787 seconds | LayerRoPE | off | 2.455527 | 2.488803 | 0.033276 |
| 787 seconds | LayerRoPE | on | 2.446882 | 2.474013 | 0.027131 |
| 787 seconds | mHC | off | 2.737210 | 2.769360 | 0.032151 |
| 787 seconds | mHC | on | 2.665257 | 2.690940 | 0.025683 |
| 787 seconds | mHC + LayerRoPE | off | 2.775536 | 2.803514 | 0.027978 |
| 787 seconds | mHC + LayerRoPE | on | 2.774611 | 2.799791 | 0.025181 |
| 1,000 updates | Residual | off | 3.045760 | 3.068173 | 0.022413 |
| 1,000 updates | Residual | on | 3.001959 | 3.027109 | 0.025150 |
| 1,000 updates | LayerRoPE | off | 2.967212 | 2.985526 | 0.018314 |
| 1,000 updates | LayerRoPE | on | 2.946600 | 2.961727 | 0.015127 |
| 1,000 updates | mHC | off | 2.948465 | 2.972336 | 0.023871 |
| 1,000 updates | mHC | on | 2.894533 | 2.913341 | 0.018808 |
| 1,000 updates | mHC + LayerRoPE | off | 2.924327 | 2.948294 | 0.023966 |
| 1,000 updates | mHC + LayerRoPE | on | 2.920748 | 2.941780 | 0.021032 |
At 787 seconds, each architecture’s mean added V-bit damage decreases with QKNorm. At 1,000 updates it decreases for LayerRoPE, mHC and the combination, while residual’s mean increases. These differences alone do not show that a smaller cache is usable or explain why the penalty changed.
Training settings and limits of the QKNorm test
The model has 32 layers, width 128, two 64-channel attention heads, feed-forward width 352, context length 256 and the GPT-2 vocabulary. Rates for residual, LayerRoPE, mHC and the combination are 0.001, 0.01, 0.003 and 0.01. Off/on pairs share common initial weights and token order. AdamW uses (0.9, 0.95), epsilon 10−8, zero decay and no clipping; warmup lasts 100 updates, then the rate stays constant.
One 80GB CUDA GPU runs one worker at a time, using Python 3.12.11, PyTorch 2.9.1+cu128, FlashAttention and four CPU threads. Master parameters, normalization, routing and optimizer state are FP32; matrix operations use BF16. TF32 and compilation are disabled. QKNorm accumulates norms in FP32 with epsilon 10−12. One unconstrained FP32 scalar per attention module is shared across heads, initialized to log2(256²−256), cast to the query dtype and multiplied into Q; attention uses scale 1.
The training clock includes batch preparation, transfers, forward/backward, optimization and scheduled checks. Setup, offline validation, attention diagnostics and checkpoint writes have separate clocks. Fresh validation and final documents exclude 184,656 recorded exact-text predecessor hashes. Validation has 32,768 targets; the final holdout has 131,072. Exclusion does not detect paraphrases. The final holdout opens only after all training statuses close.
All 24 training runs and 480 cache evaluations completed without numerical failure. The ten formats were clean K16/V16, signed K3/V16, K2/V16, K16/V3, K16/V2, K3/V3, K3/V2, K2/V3, K2/V2, and affine K2/V2. K16/V16 leaves the BF16 cache unchanged. Rounding was simulated once when each token’s cache was written; no physically packed cache or quantization-aware training was used.
mHC used its original 20-iteration normalization procedure, so this experiment did not repair or retest the stricter routing tolerance. Attention diagnostics reconstructed FP32 reference scores rather than reading the fused BF16 operation’s internal values. Three seeds, inherited learning rates and small models make this a useful lead for confirmation, not a general claim about QKNorm or a causal explanation of its effect.
Reproduction source: knlp 54ef527d. The QKNorm guide covers data preparation, numerical checks, checkpoints and Make presets.
A better model can survive rounding without being less sensitive to it
The original three-run, 2,000-update experiment illustrates this distinction. Its combination model had the best final K3/V2 loss because it started with better predictions. That did not mean mHC made the cache less sensitive to removing a value bit. The equal-time experiment later reversed the sensitivity ordering while retaining the same smallest passing cache.
The key/value (KV) cache stores attention keys and values for earlier tokens. K3/V2 uses three bits for keys and two for values; K3/V3 is the matched three-bit comparison. “Signed” means a zero-centered quantizer, even when key and value bit widths differ. An affine quantizer adds an offset so its range need not be centered on zero.
| Model | Signed K3/V3 CE | Signed K3/V2 CE | Added CE: V3 → V2 | Minimum passing bytes/token | Passing cases at that cost |
|---|---|---|---|---|---|
| Residual | 2.777245 | 2.800756 | +0.023511 | 3,072 | Affine K2/V2, Signed K3/V2 |
| mHC | 2.704911 | 2.739991 | +0.035080 | 3,072 | Affine K2/V2, Signed K3/V2 |
| LayerRoPE, β=−1.0 | 2.803767 | 2.851465 | +0.047699 | 3,072 | Signed K3/V2 |
| LayerRoPE, β=−0.5 | 2.741834 | 2.766936 | +0.025102 | 3,072 | Affine K2/V2, Signed K3/V2 |
| LayerRoPE, β=0.0 | 2.737756 | 2.760342 | +0.022586 | 3,072 | Affine K2/V2, Signed K3/V2 |
| mHC + LayerRoPE, β=−1.0 | 2.866413 | 2.918012 | +0.051600 | 3,072 | Signed K3/V2 |
| mHC + LayerRoPE, β=−0.5 | 2.733109 | 2.769681 | +0.036572 | 3,072 | Affine K2/V2, Signed K3/V2 |
| mHC + LayerRoPE, β=0.0 | 2.687847 | 2.720868 | +0.033021 | 3,072 | Affine K2/V2, Signed K3/V2 |
In this equal-token experiment, adding mHC increased the extra loss from changing K3/V3 to K3/V2: from 0.023511 to 0.035080 nats for the residual baseline, and from 0.022586 to 0.033021 for beta-zero LayerRoPE. Both increases occurred in every matched run.
Even so, the combination’s final K3/V2 loss was the lowest, at 2.720868. Its better predictions before rounding outweighed its larger value-bit penalty relative to residual and standalone LayerRoPE.
All eight configurations supported the same smallest tested cache, 3,072 theoretical bytes per token, among 51 cases. K3/V2 uses 14.3% fewer bytes than K3/V3’s 3,584 and ties affine K2/V2 after metadata. A format had to add at most 0.1 nat of loss in every seed. That allowance corresponds to roughly 10.5% higher perplexity, so passing does not mean unchanged quality.
All paired penalty comparisons and floating cache formats
| Candidate minus baseline | Mean extra-damage difference | Paired differences (nats) | Lower-damage pairs |
|---|---|---|---|
| mHC minus Residual | +0.011568 | +0.011158 / +0.012547 / +0.010999 | 0/3 |
| mHC + LayerRoPE, β=−1.0 minus LayerRoPE, β=−1.0 | +0.003901 | -0.003416 / +0.003844 / +0.011275 | 1/3 |
| LayerRoPE, β=−1.0 minus Residual | +0.024187 | +0.029909 / +0.024053 / +0.018601 | 0/3 |
| mHC + LayerRoPE, β=−1.0 minus mHC | +0.016520 | +0.015334 / +0.015349 / +0.018876 | 0/3 |
| mHC + LayerRoPE, β=−0.5 minus LayerRoPE, β=−0.5 | +0.011470 | +0.004805 / +0.022659 / +0.006947 | 0/3 |
| LayerRoPE, β=−0.5 minus Residual | +0.001590 | +0.003253 / +0.002031 / -0.000513 | 1/3 |
| mHC + LayerRoPE, β=−0.5 minus mHC | +0.001492 | -0.003100 / +0.012142 / -0.004566 | 2/3 |
| mHC + LayerRoPE, β=0.0 minus LayerRoPE, β=0.0 | +0.010435 | +0.020322 / +0.005814 / +0.005169 | 0/3 |
| LayerRoPE, β=0.0 minus Residual | -0.000926 | -0.004202 / +0.002671 / -0.001247 | 2/3 |
| mHC + LayerRoPE, β=0.0 minus mHC | -0.002059 | +0.004962 / -0.004062 / -0.007077 | 2/3 |
Scaled FP4 K4/V4 and FP8 K8/V8 pass all seeds in every row, but use more theoretical cache bytes than the smallest passing format.
| Model | FP4 K4/V4 damage | FP8 K8/V8 damage |
|---|---|---|
| Residual | +0.002329 (3/3) | +0.000116 (3/3) |
| mHC | +0.002380 (3/3) | +0.000098 (3/3) |
| LayerRoPE, β=−1.0 | +0.005847 (3/3) | +0.000215 (3/3) |
| LayerRoPE, β=−0.5 | +0.002305 (3/3) | +0.000057 (3/3) |
| LayerRoPE, β=0.0 | +0.001851 (3/3) | -0.000004 (3/3) |
| mHC + LayerRoPE, β=−1.0 | +0.007276 (3/3) | +0.000355 (3/3) |
| mHC + LayerRoPE, β=−0.5 | +0.003190 (3/3) | +0.000196 (3/3) |
| mHC + LayerRoPE, β=0.0 | +0.001831 (3/3) | +0.000151 (3/3) |
Four-bit weights fared better than four-bit residual states
We also rounded weights, block inputs and the running residual state separately in the original equal-token models. The choice of where to round mattered much more than a blanket label such as “four-bit model.” FP4 used scaled E2M1; FP8 used scaled E4M3FN. Both simulations rounded values and converted them back for computation, without retraining. These are not measurements of packed inference speed or the block-scaled NVFP4 format.
| Model | All-matrix FP4 weights | Block W4A4 | Residual-state FP4 |
|---|---|---|---|
| Residual | +0.06923 · 3/3 | +0.04008 · 3/3 | +1.07459 · 0/3 |
| LayerRoPE, β=0 | +0.05917 · 3/3 | +0.03650 · 3/3 | +2.56433 · 0/3 |
| mHC | +0.06820 · 3/3 | +0.03069 · 3/3 | +0.18705 · 0/3 |
| mHC + LayerRoPE, β=0 | +0.05909 · 3/3 | +0.02030 · 3/3 | +0.23462 · 0/3 |
Adding beta-zero LayerRoPE to mHC reduced the loss added by four-bit weights and by four-bit weights plus block inputs. Both models already passed those tests. Repeatedly rounding the residual state was much harsher: no configuration kept its loss within the 0.1-nat allowance in every seed. Eight-bit residual rounding passed for all configurations.
W4A4 means four-bit block weights and four-bit inputs to block linear operations. Residual-state rounding happens after each attention and feed-forward update, separately for every mHC stream. The weight tests leave routing, normalization and gain parameters at their original precision.
All fourteen weight, activation and residual policies
| Model | Block FP4 weights | All FP4 weights | Block inputs FP4 | Residual states FP4 | Block W4A4 | Block INT4 weights | All INT4 weights |
|---|---|---|---|---|---|---|---|
| Residual | +0.00533 (3/3) | +0.06923 (3/3) | +0.03455 (3/3) | +1.07459 (0/3) | +0.04008 (3/3) | +0.00660 (3/3) | +0.06804 (3/3) |
| mHC | +0.00400 (3/3) | +0.06820 (3/3) | +0.02751 (3/3) | +0.18705 (0/3) | +0.03069 (3/3) | +0.00516 (3/3) | +0.06827 (3/3) |
| LayerRoPE, β=−1.0 | +0.00468 (3/3) | +0.09820 (2/3) | +0.02692 (3/3) | +0.55927 (0/3) | +0.03377 (3/3) | +0.00393 (3/3) | +0.07281 (3/3) |
| LayerRoPE, β=−0.5 | +0.00421 (3/3) | +0.07000 (3/3) | +0.02981 (3/3) | +1.89288 (0/3) | +0.03351 (3/3) | +0.00699 (3/3) | +0.07498 (2/3) |
| LayerRoPE, β=0.0 | +0.00361 (3/3) | +0.05917 (3/3) | +0.03331 (3/3) | +2.56433 (0/3) | +0.03650 (3/3) | +0.00527 (3/3) | +0.09389 (2/3) |
| mHC + LayerRoPE, β=−1.0 | +0.00615 (3/3) | +0.08144 (2/3) | +0.02801 (3/3) | +0.26177 (0/3) | +0.03593 (3/3) | +0.00655 (3/3) | +0.05245 (3/3) |
| mHC + LayerRoPE, β=−0.5 | +0.00457 (3/3) | +0.06176 (3/3) | +0.02940 (3/3) | +0.50631 (0/3) | +0.03407 (3/3) | +0.00600 (3/3) | +0.06970 (3/3) |
| mHC + LayerRoPE, β=0.0 | +0.00340 (3/3) | +0.05909 (3/3) | +0.01624 (3/3) | +0.23462 (0/3) | +0.02030 (3/3) | +0.00326 (3/3) | +0.07576 (2/3) |
| Model | Block FP8 weights | All FP8 weights | Block inputs FP8 | Residual states FP8 | Block W8A8 | Block INT8 weights | All INT8 weights |
|---|---|---|---|---|---|---|---|
| Residual | +0.00049 (3/3) | +0.00329 (3/3) | +0.00095 (3/3) | +0.01463 (3/3) | +0.00134 (3/3) | -0.00012 (3/3) | +0.00008 (3/3) |
| mHC | +0.00019 (3/3) | +0.00274 (3/3) | +0.00061 (3/3) | +0.00511 (3/3) | +0.00104 (3/3) | -0.00003 (3/3) | +0.00019 (3/3) |
| LayerRoPE, β=−1.0 | +0.00007 (3/3) | +0.00275 (3/3) | +0.00059 (3/3) | +0.02300 (3/3) | +0.00078 (3/3) | -0.00005 (3/3) | +0.00004 (3/3) |
| LayerRoPE, β=−0.5 | +0.00010 (3/3) | +0.00234 (3/3) | +0.00065 (3/3) | +0.02735 (3/3) | +0.00062 (3/3) | -0.00002 (3/3) | +0.00019 (3/3) |
| LayerRoPE, β=0.0 | -0.00031 (3/3) | +0.00208 (3/3) | +0.00067 (3/3) | +0.01870 (3/3) | +0.00044 (3/3) | +0.00000 (3/3) | +0.00022 (3/3) |
| mHC + LayerRoPE, β=−1.0 | +0.00016 (3/3) | +0.00209 (3/3) | +0.00065 (3/3) | +0.01757 (3/3) | +0.00077 (3/3) | +0.00004 (3/3) | +0.00011 (3/3) |
| mHC + LayerRoPE, β=−0.5 | +0.00014 (3/3) | +0.00329 (3/3) | +0.00065 (3/3) | +0.01304 (3/3) | +0.00073 (3/3) | +0.00007 (3/3) | +0.00049 (3/3) |
| mHC + LayerRoPE, β=0.0 | +0.00011 (3/3) | +0.00172 (3/3) | +0.00039 (3/3) | +0.00429 (3/3) | +0.00056 (3/3) | +0.00005 (3/3) | -0.00011 (3/3) |
All-matrix FP4 passes only two of three seeds for both β=−1 rows; their matched INT4 tests pass all three. The result depends on the format and scope. It does not establish a general precision advantage for LayerRoPE.
Parameter coverage and ideal whole-model weight bytes
| Model | Block weight coverage | All-matrix weight coverage | Block FP4 total bytes | All FP4 total bytes | All FP8 total bytes |
|---|---|---|---|---|---|
| Residual | 33.28% | 99.96% | 54,879,744 | 10,251,528 | 19,895,688 |
| mHC | 31.92% | 95.88% | 58,163,456 | 13,535,240 | 23,179,400 |
| LayerRoPE, β=−1.0 | 33.30% | 100.00% | 54,849,088 | 10,220,872 | 19,865,032 |
| LayerRoPE, β=−0.5 | 33.30% | 100.00% | 54,849,088 | 10,220,872 | 19,865,032 |
| LayerRoPE, β=0.0 | 33.30% | 100.00% | 54,849,088 | 10,220,872 | 19,865,032 |
| mHC + LayerRoPE, β=−1.0 | 31.94% | 95.91% | 58,132,800 | 13,504,584 | 23,148,744 |
| mHC + LayerRoPE, β=−0.5 | 31.94% | 95.91% | 58,132,800 | 13,504,584 | 23,148,744 |
| mHC + LayerRoPE, β=0.0 | 31.94% | 95.91% | 58,132,800 | 13,504,584 | 23,148,744 |
Block-only rounding covers about one third of parameters and leaves the rest in FP32. Its nominal bit width is not the whole-model bit width. The routed models retain more unquantized parameters, so equal matrix precision does not imply equal total storage.
More routing iterations reduced error, but did not supply a fix
mHC normalizes the matrix that carries information between residual streams. Ideally, every row and column sums to one. Its finite sequence of normalization steps only approximates that condition. After the training runs exceeded our chosen tolerance, we replayed saved inputs to see whether more iterations or higher precision would be enough.
We tested 20, 40, 80, 160 and 320 iterations in FP32 and FP64 on the same recorded router inputs. More iterations helped, but none met the tolerance on the late checkpoints. Replaying the original calculation first reproduced its routing matrices and validation losses exactly.
| Checkpoint | Max row error, 20 | Max row error, 320 | Matrices above limit, 20 | Matrices above limit, 320 |
|---|---|---|---|---|
| mHC | 0.07641768 | 0.00323939 | 70.093% | 28.926% |
| mHC + LayerRoPE | 0.08160710 | 0.00297618 | 61.715% | 16.611% |
At 320 iterations, the worst row-sum errors were still about 30–32 times our 0.0001 limit. FP64 gave nearly the same errors as FP32, pointing to incomplete normalization rather than ordinary rounding precision as the main source on these inputs. The percentages in the table count routing matrices above the limit; they are not prediction-error rates.
What the replay covered
We replayed mHC and the combination at initialization, 200 and 787 seconds, using one qualification seed: 3,145,728 correlated matrix observations. The 200-second states passed with more iterations, but neither late state passed by 320. Sampled gradient and finite-difference checks passed. This tested saved inputs, not a whole model retrained in FP64, and the matrix count is not a count of independent training runs.
This was our stricter numerical requirement, not a broken upstream promise. The mHC paper uses 20 iterations as a practical approximation and discusses remaining deviations. The reference implementation also includes an epsilon and ends with column normalization; its default is 10 iterations, while we explicitly used 20. It does not guarantee our 10−4 tolerance.
We stopped before retraining with a changed normalization procedure. The numerical check had found no candidate that met the requirement. It therefore supplied neither a learning improvement nor a quantization improvement, and it did not establish that the routing errors caused mHC’s higher loss.
The separate digit test found no reliable transfer defense
In a separate MNIST experiment, we asked whether these architectural changes could suppress subliminal learning. A teacher learned to recognize digits. A student then imitated only the teacher’s auxiliary outputs on random noise, without seeing digit labels. We measured how much digit accuracy the student gained above its starting point.
None of the nine LayerRoPE-versus-baseline comparisons met the preselected suppression requirement. Mean transfer changes ranged from −2.283 to +4.251 percentage points. None reached the required five-point reduction. The intervals were often wide, so the result leaves some smaller effects unresolved; it does not provide a reliable defense.
We trained fourteen configurations with ten matched seeds each: residual, Hyper-Connections (HC), mHC, their full LayerRoPE variants, and two input-gains-only controls. Each used four width-256 feed-forward blocks, five teacher epochs and 80 student epochs, FP32, and Adam at 0.0003, with LayerNorm centering and bias.
The baseline mHC and all three full LayerRoPE+mHC variants also exceeded our 10−4 routing row/column tolerance on some monitored training or control paths. The largest column error approached one. The auxiliary-only paths with shared initialization stayed below it, but the requirement covered all monitored paths. These are limits of this experiment’s numerical qualification, not evidence that transfer was suppressed.
Paired suppression comparisons and decision rule
A candidate needs at least five percentage points less mean transfer, a paired interval wholly below zero, preserved teacher and all-output accuracy within two points, acceptable convergence, and tracked routing stability. Intervals use confidence 1−0.05/9 (99.44%) and 100,000 bootstrap resamples across all ten seeds.
| Full-gain row | Transfer change (pp) | Interval (pp) | Unmet requirements |
|---|---|---|---|
| Residual + full LayerRoPE, β=−1.0 | +0.941 | [-8.662, +9.089] | transfer, speed teacher |
| Residual + full LayerRoPE, β=−0.5 | -0.888 | [-9.913, +6.018] | transfer, speed all output, speed teacher |
| Residual + full LayerRoPE, β=0.0 | +3.398 | [+0.169, +6.189] | transfer, speed all output |
| HC + full LayerRoPE, β=−1.0 | -2.283 | [-12.173, +5.608] | transfer, speed all output, speed teacher |
| HC + full LayerRoPE, β=−0.5 | +0.213 | [-6.051, +5.854] | transfer, speed all output, speed teacher |
| HC + full LayerRoPE, β=0.0 | +1.944 | [-3.125, +6.210] | transfer, speed all output |
| mHC + full LayerRoPE, β=−1.0 | +0.806 | [-9.159, +9.437] | transfer, speed teacher, stability |
| mHC + full LayerRoPE, β=−0.5 | -0.437 | [-8.519, +5.868] | transfer, speed all output, speed teacher, stability |
| mHC + full LayerRoPE, β=0.0 | +4.251 | [+0.962, +7.380] | transfer, speed all output, stability |
Full MNIST accuracy, convergence and quantization endpoints
| Model | Teacher accuracy (%) | Auxiliary-only gain (pp) | All-output accuracy (%) | Epochs to 95% |
|---|---|---|---|---|
| Residual | 97.285 | 60.804 | 96.714 | 13.46 |
| Residual + full LayerRoPE, β=−1.0 | 97.014 | 61.745 | 96.635 | 11.03 |
| Residual + full LayerRoPE, β=−0.5 | 97.166 | 59.916 | 96.580 | 12.94 |
| Residual + full LayerRoPE, β=0.0 | 97.211 | 64.202 | 96.655 | 12.50 |
| Residual + input gains only | 97.263 | 61.186 | 96.727 | 11.51 |
| HC | 97.287 | 65.894 | 96.727 | 11.26 |
| HC + full LayerRoPE, β=−1.0 | 97.176 | 63.611 | 96.589 | 11.75 |
| HC + full LayerRoPE, β=−0.5 | 97.227 | 66.107 | 96.635 | 11.27 |
| HC + full LayerRoPE, β=0.0 | 97.202 | 67.838 | 96.558 | 12.60 |
| HC + input gains only | 97.278 | 65.779 | 96.694 | 9.87 |
| mHC | 97.279 | 61.590 | 96.617 | 14.26 |
| mHC + full LayerRoPE, β=−1.0 | 97.066 | 62.396 | 96.647 | 11.30 |
| mHC + full LayerRoPE, β=−0.5 | 97.195 | 61.153 | 96.602 | 13.11 |
| mHC + full LayerRoPE, β=0.0 | 97.215 | 65.841 | 96.587 | 13.22 |
| Model | Clean accuracy (%) | All-weight FP4 accuracy (%) | All-weight FP4 extra CE (nats) | FP4 ideal total weight bytes | All-weight FP8 loss (pp) | Block W4A4 loss (pp) |
|---|---|---|---|---|---|---|
| Residual | 96.714 | 96.226 | +0.01829 | 392,936 | +0.032 | +0.441 |
| Residual + full LayerRoPE, β=−1.0 | 96.635 | 96.353 | +0.01170 | 390,920 | +0.007 | +0.291 |
| Residual + full LayerRoPE, β=−0.5 | 96.580 | 96.189 | +0.01651 | 390,920 | +0.027 | +0.336 |
| Residual + full LayerRoPE, β=0.0 | 96.655 | 96.367 | +0.01357 | 390,920 | +0.015 | +0.449 |
| Residual + input gains only | 96.727 | 96.371 | +0.01409 | 389,880 | +0.025 | +0.357 |
| HC | 96.727 | 96.367 | +0.01384 | 426,120 | +0.038 | +0.464 |
| HC + full LayerRoPE, β=−1.0 | 96.589 | 96.293 | +0.01101 | 424,104 | -0.008 | +0.338 |
| HC + full LayerRoPE, β=−0.5 | 96.635 | 96.319 | +0.01195 | 424,104 | +0.045 | +0.494 |
| HC + full LayerRoPE, β=0.0 | 96.558 | 96.052 | +0.01833 | 424,104 | +0.022 | +0.711 |
| HC + input gains only | 96.694 | 96.268 | +0.01646 | 423,064 | +0.033 | +0.511 |
| mHC | 96.617 | 96.224 | +0.01552 | 802,968 | +0.004 | +0.421 |
| mHC + full LayerRoPE, β=−1.0 | 96.647 | 96.363 | +0.01228 | 800,952 | +0.011 | +0.233 |
| mHC + full LayerRoPE, β=−0.5 | 96.602 | 96.330 | +0.01163 | 800,952 | +0.025 | +0.380 |
| mHC + full LayerRoPE, β=0.0 | 96.587 | 96.147 | +0.01590 | 800,952 | +0.006 | +0.528 |
All digit scores use the 10,000 test images. The packed ten-model row is not ten independent wall-clock repeats. All-output and auxiliary-only endpoints are distinct; the reproduction retains teacher and student quantization screens. There is no MNIST residual-state rounding intervention.
Give each layer a different way to read and write
A residual stream is the running state carried through a network. Each attention or feed-forward block reads that state, computes an update, and adds it back. A gain multiplies each channel: it can strengthen, weaken or reverse that channel.
LayerRoPE generates these gains from a shared learned vector and the layer number. It scales the vector’s length with a magnitude schedule and rotates pairs of gain entries with a rotation schedule. The resulting gains multiply the activations channel by channel. The rotation operates on the gain vector, rather than directly rotating the residual state or cached keys and values. See the LayerRoPE construction.
One running state
Each block reads the state and adds an update. The baseline uses ordinary learned input-normalization gains.
Depth-dependent channel gains
The input and output controls depend on the layer number. A shared gain vector replaces independent vectors at every layer.
Four residual streams
Manifold-constrained Hyper-Connections learn how to read, carry and write multiple streams. The stream mixer uses 20 alternating normalization steps (Sinkhorn iterations) to bring its row and column sums toward one. LayerRoPE can also control the block’s input and output gains inside this model.
Two beta knobs, at both input and output
βmag sets how gain magnitude changes with depth. βrot sets how the rotation angle changes with depth. They are separate parameters. The measured sweep starts both at the same value, −1, −0.5 or 0, at all four sites; training can then change each independently.
At initialization, a more negative magnitude beta makes deeper layers’ gains smaller. A more negative rotation beta makes their rotation angles fall faster with depth. Input gains control what a block reads; output gains control the update it writes back.
Move the controls to see the difference
This is the gain formula, not a trained checkpoint or a prediction of model quality. The shared pair is (1, 1), both intercepts are zero, and this first pair has frequency 1.
Zero is not “LayerRoPE off.” At β=0 the initial schedules are constant across depth, but the gains still apply, and the beta parameters remain trainable.
The exact schedules and the remaining knobs
The learned α intercepts set the overall magnitude and angular scale. The shared gain vector sets the channel pattern. The frequency base, 100 here, spreads the rotation across pairs; it is fixed in this sweep. Every input/output site has its own parameters.
With zero intercepts at layer 16, β=−1 gives magnitude 1/16, β=−0.5 gives 1/4, and β=0 gives 1. The first pair’s angles are respectively 1/16, 1/4 and 1 radian. These examples describe initialization. They do not determine the learned residual variance or guarantee better quantization.
Residual variance did not explain the quantization results
One possible explanation was that LayerRoPE changes the size and distribution of residual values in a way that makes them easier to round. But changing their overall magnitude can be absorbed by the quantizer’s scale. We need to see less relative rounding error or better final predictions to call it useful.
The negative-beta models actually had lower final residual variance here. Beta-zero LayerRoPE varied widely between seeds. These observations do not reproduce a general widening of the residual stream, and the experiment did not isolate magnitude, rotation or output gains to determine which caused a change.
Residual distributions and weight statistics for all eight configurations
| Model | Unaveraged variance by seed | Collapsed-mean variance by seed | Mean relative FP4 MSE | Mean relative FP8 MSE |
|---|---|---|---|---|
| Residual | 2459 / 2695 / 2336 | single stream | 0.01639 | 0.000592 |
| mHC | 1669 / 1705 / 1427 | 1668 / 1702 / 1426 | 0.01630 | 0.000591 |
| LayerRoPE, β=−1.0 | 5.583 / 5.191 / 5.359 | single stream | 0.02251 | 0.000515 |
| LayerRoPE, β=−0.5 | 32.07 / 32.39 / 259.1 | single stream | 0.02228 | 0.000524 |
| LayerRoPE, β=0.0 | 1.587e+04 / 10.43 / 6.799 | single stream | 0.02297 | 0.000510 |
| mHC + LayerRoPE, β=−1.0 | 5.523 / 5.431 / 5.43 | 5.523 / 5.431 / 5.43 | 0.01686 | 0.000584 |
| mHC + LayerRoPE, β=−0.5 | 49.75 / 55.96 / 288.4 | 49.75 / 55.95 / 288.4 | 0.02361 | 0.000503 |
| mHC + LayerRoPE, β=0.0 | 21.02 / 20.45 / 15.13 | 21.01 / 20.42 / 15.12 | 0.02038 | 0.000541 |
| Model | Attention weight RMS | MLP weight RMS | Attention relative FP4 MSE | MLP relative FP4 MSE |
|---|---|---|---|---|
| Residual | 0.10579 | 0.10793 | 0.01186 | 0.01232 |
| mHC | 0.11040 | 0.11901 | 0.01183 | 0.01216 |
| LayerRoPE, β=−1.0 | 0.35733 | 0.35708 | 0.01192 | 0.01245 |
| LayerRoPE, β=−0.5 | 0.41218 | 0.40907 | 0.01192 | 0.01234 |
| LayerRoPE, β=0.0 | 0.38805 | 0.35035 | 0.01192 | 0.01250 |
| mHC + LayerRoPE, β=−1.0 | 0.12262 | 0.13236 | 0.01182 | 0.01220 |
| mHC + LayerRoPE, β=−0.5 | 0.41572 | 0.40725 | 0.01188 | 0.01228 |
| mHC + LayerRoPE, β=0.0 | 0.39519 | 0.39012 | 0.01189 | 0.01219 |
RMS is root-mean-square magnitude; MSE is mean squared rounding error. Weight magnitudes are also affected by the row-specific learning rate. Unaveraged routed streams and their collapsed mean are measured separately.
What the fixed-token measurements cover
The language models have 32 layers, width 128, two attention heads, feed-forward width 352, a 256-token context and the GPT-2 vocabulary. Residual/LayerRoPE models have about 19.3 million parameters; mHC variants have about 20.1 million. Each final model receives 2,000 updates of 2,048 predicted tokens. Each row first receives four 500-update learning-rate pilots at 0.0003, 0.001, 0.003 and 0.01. Mean pilot CE at updates 300/400/500 selects the rate; ties select the smaller rate.
Final seeds 77311–77313 pair common initial matrices and training-token orders across configurations. Every model starts fresh; none continues an earlier trained checkpoint. The data preparation and evaluation arrays are reused with hash checks, so the three-seed results are descriptive evidence, not independent confirmation.
Measurements use two matched 80GB GPUs, Python 3.12.11, PyTorch 2.9.1 and four CPU threads per worker. Master weights, optimizer state, normalization, gains, routing and CE use FP32. Language matrix operations use BF16; TF32 and compilation are disabled. AdamW uses (0.9, 0.95), epsilon 10−8, no weight decay or clipping, and 100 warm-up updates followed by a constant learning rate.
Quantization runs after training on unchanged final checkpoints. Each receives fourteen weight/input/residual policies and 51 cache cases: clean, the 16/8/4/3/2-bit K/V grid for signed and affine quantizers, and FP4/FP8 cache cases. Scales and metadata enter ideal byte counts. No native packed-kernel, measured serving-memory or throughput benefit follows from these simulations.
Measured source: knlp 17035f84. The experiment protocol defines the gains, controls, quantization scopes and retained checkpoints.
Reproduce the comparisons
You can reproduce the experiments through knlp’s standard make defconfig-…; make workflow. Start with the CPU smoke check for the experiment you want to run. It exercises the code on synthetic inputs; reproducing the reported learning results requires the full data and GPU training.
make defconfig-layerrope-mhc-beta-smoke
make
The full beta comparison needs an existing two-GPU allocation and the documented runtime. It runs numerical checks, learning-rate selection, language and MNIST training, and quantization evaluation. The commands do not rent compute.
make defconfig-layerrope-mhc-beta
make CONFIG_RESIDUAL_OUTPUT_DIR=/path/to/runs \
CONFIG_RESIDUAL_CHECKPOINT_DIR=/path/to/checkpoints \
CONFIG_RESIDUAL_CPU_PYTHON=/path/to/python
Install dependencies from the reproduction guide. Use make menuconfig for external data/cache locations and runtime settings. Language checkpoints retain model, optimizer and random-number-generator states at updates 0, 1000 and 2000; pilot endpoints and MNIST teacher/student states are also retained. Generated reports, arrays, logs and checkpoints stay outside the source checkout.
Equal-training-time comparison
The time-budget presets train the four architectures with the same clock and evaluate the saved checkpoints. The time-budget guide explains how to prepare the data, run numerical checks and set explicit resource limits.
make defconfig-layerrope-mhc-time-budget-smoke
make
make defconfig-layerrope-mhc-time-budget
make CONFIG_RESIDUAL_TIME_CARD=/path/to/CARD.json \
CONFIG_RESIDUAL_TIME_RESOURCE=/path/to/RESOURCE.json \
CONFIG_RESIDUAL_TIME_ACTION=qualify
QKNorm: training and saved-checkpoint evaluation
The QKNorm CPU smoke check covers all four architectures with normalization off and on, plus the ten cache formats. It checks attention, gradients, matched initialization, resume and cache behavior.
make defconfig-layerrope-mhc-qknorm-smoke
make CONFIG_RESIDUAL_OUTPUT_DIR=/path/to/qknorm-smoke
The QKNorm reproduction guide covers disjoint data splits, seed selection and the experiment configuration. Full training needs one existing CUDA GPU, a passing qualification run and explicit time limits. The final evaluation data stays separate from those checks.
make defconfig-layerrope-mhc-qknorm
make CONFIG_RESIDUAL_TIME_ACTION=qualify \
CONFIG_RESIDUAL_TIME_CARD=/path/to/CARD.json \
CONFIG_RESIDUAL_TIME_RESOURCE=/path/to/QUALIFY-RESOURCE.json \
CONFIG_RESIDUAL_OUTPUT_DIR=/path/to/qualification
make CONFIG_RESIDUAL_TIME_ACTION=run \
CONFIG_RESIDUAL_TIME_CARD=/path/to/CARD.json \
CONFIG_RESIDUAL_TIME_RESOURCE=/path/to/RUN-RESOURCE.json \
CONFIG_RESIDUAL_OUTPUT_DIR=/path/to/results
The run configuration references the passing qualification result and its hash. Training saves complete state at 1,000 updates and at approximately 787 training seconds. To evaluate saved models without retraining, point the cache preset at the existing qknorm/ directory containing training/ and evaluation/. It reuses verified results and evaluates any missing cases:
make defconfig-layerrope-mhc-qknorm-kv-cache
make CONFIG_RESIDUAL_TIME_CARD=/path/to/CARD.json \
CONFIG_RESIDUAL_TIME_RESOURCE=/path/to/CACHE-RESOURCE.json \
CONFIG_RESIDUAL_CHECKPOINT_DIR=/path/to/existing/qknorm \
CONFIG_RESIDUAL_OUTPUT_DIR=/path/to/cache-receipts
The reports include final prediction loss, loss added by rounding, key-only and value-bit effects, and cache sizes under both quality requirements. Reports, arrays and checkpoints are written outside the source checkout.
Finite-routing replay
The routing smoke check uses synthetic inputs. Replaying the reported diagnostic requires the six saved qualification checkpoints and an input manifest. The routing-input guide provides data preparation commands and a manifest generator for your own checkpoints. Newly trained checkpoints make a new experiment; a different runtime may not reproduce the original losses exactly.
make defconfig-layerrope-mhc-routing-audit-smoke
make
python -m research.layerrope_beta.time_budget.routing_inputs manifest \
--data /path/to/training \
--result /path/to/M/RESULT.json --result /path/to/ML/RESULT.json \
--out /path/to/AUDIT-INPUTS.json
make defconfig-layerrope-mhc-routing-audit
make CONFIG_RESIDUAL_AUDIT_INPUTS=/path/to/AUDIT-INPUTS.json
The routing command only runs the diagnostic. It does not launch another training experiment. Generated inputs and results stay outside the source checkout.
Read the methods
- LayerRoPE: Dynamic Depth-wise Magnitude & Angular Superposition (Audio Summary, Visualization) — depth-conditioned gains and their construction.
- mHC: Manifold-Constrained Hyper-Connections (Audio Summary) — finite constrained routing of multiple residual streams.
- Query-Key Normalization for Transformers — per-head query/key normalization with learned attention-score strength.
- Training and evaluation code — the measured beta matrix and fake-quantization policies.
- MNIST transfer study — background on the separate teacher/student experiment.