Learning efficiency · KV cache precision

LayerRoPE and mHC.
What the experiments showed.

We tested whether changing a model’s residual connections could improve learning and make its key/value (KV) cache easier to compress. On these small language models, learning more from each token did not mean learning more in the same time. Ordinary residual connections finished with the best predictions when every model had the same time budget.

What we learned

Same training time
Residual

The simple baseline finished with the lowest loss, before and after K3/V2 cache rounding. It completed far more updates. LayerRoPE came second; adding mHC did not earn its runtime cost here. See the eight-seed comparison →

Same training tokens
Combination

With every model limited to 2,000 updates, mHC + LayerRoPE had the lowest mean loss. That earlier result showed better use of training data, but each update took much longer. See the original comparison →

Smaller KV cache
No extra gain

Each architecture’s smallest tested cache stayed at 3,072 theoretical bytes per token when rounding could add at most 0.1 nat of loss. QKNorm did not change that result. See the two quality tests →

The most promising follow-up was QKNorm. Normalizing attention’s queries and keys improved mHC in all three matched runs at equal time. It still did not catch the residual baseline. A smaller-cache result appeared at equal updates under a shared quality target; it disappeared when training time was held equal. The QKNorm results explain why.

These are 32-layer, roughly 20-million-parameter TinyStories models with 256-token contexts. The conclusions describe this implementation and training budget. Cache sizes include scale metadata but are calculated: the experiments simulated low-precision rounding, without packing a smaller serving cache.

How to read the results

Loss measures how well the model predicts the next token; lower is better. Tables use cross-entropy (CE), in nats. Clean means evaluation without the extra quantization being tested. Perplexity is exp(CE).

K3/V2 means three bits for cached keys and two for cached values. Comparing it with K3/V3 measures the extra loss from removing one value bit. A model can lose less from that change and still finish worse than a stronger model.

A seed sets the random initialization and training order. Paired runs share those choices across models. The equal-token, equal-time and QKNorm experiments use separate seeds and evaluation sets, so compare models within each experiment rather than treating their losses as one continuous training curve.

The combination learned more from the same training data

Tables scroll horizontally on narrow screens.

We began with a simple comparison: give every model the same 4.096 million training tokens and see which learns most. We tested ordinary residual connections, mHC, and three initial settings for LayerRoPE, both alone and combined with mHC. Each of the eight configurations had the same learning-rate search budget and three matched training runs.

The combination with LayerRoPE’s beta parameters initialized to zero had the lowest mean prediction loss. Beta controls how its gains vary with layer depth; zero is a starting setting, not a switch that turns LayerRoPE off. The interactive explanation shows the difference.

Original equal-token experiment: three paired final seeds, 2,000 updates and 4.096 million predicted training tokens per model. Average validation CE is measured at updates 1000–2000. Clean final CE uses the separate 131,072-token final array. PPL is exp(mean clean CE). Training seconds sum the optimizer steps and exclude curve evaluation, statistics, checkpoint I/O and quantization.
ModelSelected LRParameters (M)Average validation CEFinal clean CEPPLTraining seconds
Residual0.00319.2972.6901082.76503615.880121.1
mHC0.00320.1182.6195912.69252614.769615.0
LayerRoPE, β=−1.00.0119.2892.7092332.77220715.994231.8
LayerRoPE, β=−0.50.0119.2892.6672772.72919315.321250.4
LayerRoPE, β=0.00.0119.2892.6517012.72762715.297254.4
mHC + LayerRoPE, β=−1.00.00320.1102.7668452.82979216.942826.4
mHC + LayerRoPE, β=−0.50.0120.1102.6548102.71557615.113786.7
mHC + LayerRoPE, β=0.00.0120.1102.6024042.67680414.539786.8
The initial result

Adding beta-zero LayerRoPE to mHC reduced average validation loss by 0.017187 nats and final clean loss by 0.015722, with improvement in all three matched runs. That was a real lead in this experiment. The unanswered question was whether it justified the extra training time.

Each configuration selected its own learning rate from the same four candidates. This compares tuned recipes, so the differences cannot be assigned to the architecture alone. Several selected the highest rate tested, 0.01; a wider search might change the result.

Validation curves for all beta initializations
Residual and full LayerRoPE: mean validation cross-entropy and the three-seed range 100 500 1000 1500 2000 Optimizer updates 2.5 3.0 3.5 4.0 4.5 5.0 Curve cross-entropy (nats) Residual and full LayerRoPE Residual β=−1.0 β=−0.5 β=0.0
mHC and full LayerRoPE: mean validation cross-entropy and the three-seed range 100 500 1000 1500 2000 Optimizer updates 2.5 3.0 3.5 4.0 4.5 5.0 Curve cross-entropy (nats) mHC and full LayerRoPE mHC β=−1.0 β=−0.5 β=0.0
Mean curve cross-entropy over three fresh paired seeds. Bands show the seed minimum and maximum, not confidence intervals. Curves begin at update 100, after warm-up; shading marks the preselected averaging window, updates 1000–2000. Each row uses its pilot-selected learning rate. These plots compare loss per update; training seconds are reported separately.

At the same training time, ordinary residual connections won

We then gave each model 787 seconds—about 13 minutes—of training. The residual baseline completed about 11,534 updates, compared with 2,076 for mHC and 1,655 for the combination. That extra training paid off: residual finished with the lowest mean loss, and standalone LayerRoPE was second.

This comparison used eight new matched runs per model, a fresh evaluation set and the same five-candidate learning-rate search budget for every architecture. It did not resume the earlier models. Both LayerRoPE beta parameters started at zero and remained trainable. In the table, clean means no added rounding; K3/V2 uses three-bit keys and two-bit values.

Means over eight paired seeds at 787 synchronized training seconds. Every endpoint predicts the same 131,072 fresh held-out next tokens. Cross-entropy (CE) is in nats per token; lower is better. K3/V3 and K3/V2 use the same signed quantizer with different value bit widths.
ModelClean CEK3/V3 CEK3/V2 CEK3/V2 − K3/V3Mean updates
Residual2.3547922.3800442.442863+0.06281911,534.000
LayerRoPE, β=02.4837802.4981852.533388+0.0352035,438.125
mHC2.7711972.7833442.813972+0.0306272,076.250
mHC + LayerRoPE, β=02.8281592.8376632.866000+0.0283381,655.125

Residual and LayerRoPE each beat both mHC versions in all eight matched runs, both before rounding and with a K3/V2 cache. All eight corresponding comparisons also passed the preselected 0.01-nat improvement rule for the median paired difference. Residual had the lowest observed mean; a formal residual-versus-LayerRoPE claim was outside that test family. The protocol gives the statistical details.

mHC lost less from the extra value-bit rounding, but still produced worse predictions. Its K3/V2 loss was 2.813972, versus 2.442863 for residual. This is why a small quantization penalty alone is not enough: what matters is the quality of the model you end up using.

All four architectures supported the same smallest tested cache: 3,072 theoretical bytes per token, including metadata, with no more than 0.1 nat of added loss in every run. None supported the smaller 2,560-byte format across all eight runs.

What counted as training time?

The clock included preparing and transferring batches, warm-up, forward/backward computation, optimizer updates and scheduled numerical checks. It excluded setup, offline evaluation and checkpoint writes. Each checkpoint followed a complete update, with at most one update and 1% overshoot. This measures training efficiency; total experiment cost also includes the excluded work.

The selected learning rates were 0.001 for residual, 0.01 for LayerRoPE, 0.003 for mHC and 0.01 for the combination. These timings come from the portable implementation used here. Optimized kernels could change the tradeoff.

The mHC runs also exceeded our chosen tolerance for how accurately their routing weights should be normalized. Their values remained finite, and every run stays in the comparison. A separate numerical investigation found no qualified fix; it did not show that routing error caused the learning gap.

QKNorm helped mHC, but did not close the gap

Could the attention calculation itself help? We added Query-Key Normalization (QKNorm) to each architecture and repeated the comparison with three new matched runs, training 24 models in total. QKNorm reduced mHC’s mean clean loss from 2.722623 to 2.653771 at the same time budget, improving all three runs. Residual without QKNorm still led, at 2.280066.

QKNorm normalizes each query and key vector to unit length, then learns a scale for the attention scores. It adds no normalization to values. LayerRoPE instead controls how layers read and write their state; mHC mixes several residual streams. The changes can therefore be tested together.

We kept each architecture’s previous learning rate when switching QKNorm on. This was a focused test of that change, not a new search for the best settings. Its three runs are separate from the eight-run equal-time experiment above.

Project Q and Ksplit into attention heads
Normalize Q and Kuse vector direction
Positional RoPEencode token position
Learn score strengthone scalar per layer
Attention over Vno added V normalization
This diagram describes the QKNorm-on path. The scalar multiplies Q before the attention dot product. New cache entries store the normalized, position-rotated K and the projected V without extra normalization, then apply the selected rounding once. Existing cached entries are not normalized or rounded again.

The benefit depends on what you hold fixed

Switch between 787 seconds of training and 1,000 updates. Each update processes 2,048 training tokens. Both checkpoints are scored on the same 131,072 held-out targets. Lower bars mean better predictions.

787 training seconds · mean clean CE · QKNorm off versus on

Residual

Off2.280
On2.292

On minus off: +0.011829 nats

LayerRoPE

Off2.441
On2.433

On minus off: −0.008396 nats

mHC

Off2.723
On2.654

On minus off: −0.068852 nats

mHC + LayerRoPE

Off2.766
On2.765

On minus off: −0.001448 nats

All bars use the same zero-based 0–3.2 nat scale. The full table below remains available without JavaScript. Means and paired ranges describe these three seeds; ranges are not confidence intervals.

Three paired seeds, 77811–77813. Delta is QKNorm on minus off; negative favors QKNorm. LayerRoPE uses full input/output magnitude and rotation, with both beta slopes initialized to zero and trainable. The learning rates are inherited from the QKNorm-off study, with no equal tuning claim.
EndpointArchitectureOff clean CEOn clean CEMean deltaPaired delta range
787 secondsResidual2.2800662.291895+0.011829+0.001674 to +0.018557
787 secondsLayerRoPE2.4410222.432625−0.008396−0.018279 to +0.004146
787 secondsmHC2.7226232.653771−0.068852−0.085912 to −0.057450
787 secondsmHC + LayerRoPE2.7661672.764719−0.001448−0.023085 to +0.012058
1,000 updatesResidual3.0397822.991183−0.048599−0.059928 to −0.036639
1,000 updatesLayerRoPE2.9605152.938223−0.022292−0.033617 to −0.005758
1,000 updatesmHC2.9389372.886090−0.052847−0.069344 to −0.021662
1,000 updatesmHC + LayerRoPE2.9175022.912387−0.005116−0.039031 to +0.024791

At 1,000 updates, QKNorm improved residual, LayerRoPE and mHC in every matched run; mHC with QKNorm had the lowest mean loss. But QKNorm also made the residual baseline slower. Within 787 seconds it completed 8,276–8,922 updates, versus 9,999–11,371 without QKNorm, and finished worse in all three pairs. Better updates did not compensate for fewer updates. LayerRoPE and the combination showed mixed results at equal time.

Did QKNorm buy a smaller cache?

At equal training time, no. The answer at equal updates needs more care, because we tested two different quality requirements:

How much did rounding hurt this model?

Compare the quantized model with its own unrounded version. Allow at most 0.1 nat of added loss. Under this test, every architecture still needed at least 3,072 theoretical bytes per cached token, with QKNorm off or on, at both checkpoints.

Is the final model good enough?

Compare every model with the same unrounded residual baseline, allowing 0.1 nat above its loss. A better starting model can spend some of that quality advantage on stronger rounding. This tests the final result, including both training quality and quantization damage.

The table uses the second test: one shared quality target, set by the paired residual model without QKNorm at the same checkpoint. One fixed cache format must pass every seed. “None” means none of the tested formats met that target, even before rounding.

Ideal bytes per cached token, including scales and offsets. Each arrow is QKNorm off → on. “None” means no tested policy qualifies, including clean. Actual cache tensors remain BF16; these are not measured serving-memory savings.
Architecture787 seconds1,000 updates
Residual3,072 → 3,0723,072 → 3,072
LayerRoPENone → None2,560 → 2,560
mHCNone → None3,072 → 2,560
mHC + LayerRoPENone → None2,560 → 2,560

At 1,000 updates, QKNorm let mHC use signed K2/V2 and still meet the shared target: 2,560 instead of 3,072 theoretical bytes per token, a 16.7% reduction. Its better starting quality helped make that possible. The tolerated damage relative to its own unrounded version did not improve enough to change that model-relative minimum. LayerRoPE and the combination already passed at 2,560 bytes without QKNorm.

At equal time, only residual met the shared target, at 3,072 bytes with or without QKNorm. The other architectures already missed it before cache rounding. The equal-update saving therefore did not become an equal-time advantage.

K3/V3 versus K3/V2: inspect both endpoints

K3/V3 is the reference for lowering V alone from three to two bits while holding K at three. Both use the same signed codebook rule; unequal K/V widths are a separate choice from signed versus affine quantization. K3/V3 costs 3,584 ideal bytes per token and K3/V2 costs 3,072.

Means over the same three seeds, all 32 attention layers rounded. Added CE is K3/V2 minus K3/V3; lower means less added damage from reducing V precision. Absolute CE and this damage measure have different baselines.
EndpointArchitectureQKNormK3/V3 CEK3/V2 CEAdded CE: V3 → V2
787 secondsResidualoff2.3070412.3710750.064035
787 secondsResidualon2.3144192.3682500.053832
787 secondsLayerRoPEoff2.4555272.4888030.033276
787 secondsLayerRoPEon2.4468822.4740130.027131
787 secondsmHCoff2.7372102.7693600.032151
787 secondsmHCon2.6652572.6909400.025683
787 secondsmHC + LayerRoPEoff2.7755362.8035140.027978
787 secondsmHC + LayerRoPEon2.7746112.7997910.025181
1,000 updatesResidualoff3.0457603.0681730.022413
1,000 updatesResidualon3.0019593.0271090.025150
1,000 updatesLayerRoPEoff2.9672122.9855260.018314
1,000 updatesLayerRoPEon2.9466002.9617270.015127
1,000 updatesmHCoff2.9484652.9723360.023871
1,000 updatesmHCon2.8945332.9133410.018808
1,000 updatesmHC + LayerRoPEoff2.9243272.9482940.023966
1,000 updatesmHC + LayerRoPEon2.9207482.9417800.021032

At 787 seconds, each architecture’s mean added V-bit damage decreases with QKNorm. At 1,000 updates it decreases for LayerRoPE, mHC and the combination, while residual’s mean increases. These differences alone do not show that a smaller cache is usable or explain why the penalty changed.

Training settings and limits of the QKNorm test

The model has 32 layers, width 128, two 64-channel attention heads, feed-forward width 352, context length 256 and the GPT-2 vocabulary. Rates for residual, LayerRoPE, mHC and the combination are 0.001, 0.01, 0.003 and 0.01. Off/on pairs share common initial weights and token order. AdamW uses (0.9, 0.95), epsilon 10−8, zero decay and no clipping; warmup lasts 100 updates, then the rate stays constant.

One 80GB CUDA GPU runs one worker at a time, using Python 3.12.11, PyTorch 2.9.1+cu128, FlashAttention and four CPU threads. Master parameters, normalization, routing and optimizer state are FP32; matrix operations use BF16. TF32 and compilation are disabled. QKNorm accumulates norms in FP32 with epsilon 10−12. One unconstrained FP32 scalar per attention module is shared across heads, initialized to log2(256²−256), cast to the query dtype and multiplied into Q; attention uses scale 1.

The training clock includes batch preparation, transfers, forward/backward, optimization and scheduled checks. Setup, offline validation, attention diagnostics and checkpoint writes have separate clocks. Fresh validation and final documents exclude 184,656 recorded exact-text predecessor hashes. Validation has 32,768 targets; the final holdout has 131,072. Exclusion does not detect paraphrases. The final holdout opens only after all training statuses close.

All 24 training runs and 480 cache evaluations completed without numerical failure. The ten formats were clean K16/V16, signed K3/V16, K2/V16, K16/V3, K16/V2, K3/V3, K3/V2, K2/V3, K2/V2, and affine K2/V2. K16/V16 leaves the BF16 cache unchanged. Rounding was simulated once when each token’s cache was written; no physically packed cache or quantization-aware training was used.

mHC used its original 20-iteration normalization procedure, so this experiment did not repair or retest the stricter routing tolerance. Attention diagnostics reconstructed FP32 reference scores rather than reading the fused BF16 operation’s internal values. Three seeds, inherited learning rates and small models make this a useful lead for confirmation, not a general claim about QKNorm or a causal explanation of its effect.

Reproduction source: knlp 54ef527d. The QKNorm guide covers data preparation, numerical checks, checkpoints and Make presets.

A better model can survive rounding without being less sensitive to it

The original three-run, 2,000-update experiment illustrates this distinction. Its combination model had the best final K3/V2 loss because it started with better predictions. That did not mean mHC made the cache less sensitive to removing a value bit. The equal-time experiment later reversed the sensitivity ordering while retaining the same smallest passing cache.

The key/value (KV) cache stores attention keys and values for earlier tokens. K3/V2 uses three bits for keys and two for values; K3/V3 is the matched three-bit comparison. “Signed” means a zero-centered quantizer, even when key and value bit widths differ. An affine quantizer adds an offset so its range need not be centered on zero.

Value-precision penalty = CE(K3/V2) − CE(K3/V3) Architecture comparison = candidate’s penalty − baseline’s penalty Negative favors the candidate’s tolerance to lowering value precision.
All 32 attention layers use the stated cache quantizer. K3/V3 costs 3,584 ideal bytes/token; K3/V2 costs 3,072. The last columns give the smallest byte cost among all 51 cache cases with damage ≤0.1 nats from own clean in every seed. This engineering tolerance permits loss and does not mean unchanged quality.
ModelSigned K3/V3 CESigned K3/V2 CEAdded CE: V3 → V2Minimum passing bytes/tokenPassing cases at that cost
Residual2.7772452.800756+0.0235113,072Affine K2/V2, Signed K3/V2
mHC2.7049112.739991+0.0350803,072Affine K2/V2, Signed K3/V2
LayerRoPE, β=−1.02.8037672.851465+0.0476993,072Signed K3/V2
LayerRoPE, β=−0.52.7418342.766936+0.0251023,072Affine K2/V2, Signed K3/V2
LayerRoPE, β=0.02.7377562.760342+0.0225863,072Affine K2/V2, Signed K3/V2
mHC + LayerRoPE, β=−1.02.8664132.918012+0.0516003,072Signed K3/V2
mHC + LayerRoPE, β=−0.52.7331092.769681+0.0365723,072Affine K2/V2, Signed K3/V2
mHC + LayerRoPE, β=0.02.6878472.720868+0.0330213,072Affine K2/V2, Signed K3/V2

In this equal-token experiment, adding mHC increased the extra loss from changing K3/V3 to K3/V2: from 0.023511 to 0.035080 nats for the residual baseline, and from 0.022586 to 0.033021 for beta-zero LayerRoPE. Both increases occurred in every matched run.

Even so, the combination’s final K3/V2 loss was the lowest, at 2.720868. Its better predictions before rounding outweighed its larger value-bit penalty relative to residual and standalone LayerRoPE.

All eight configurations supported the same smallest tested cache, 3,072 theoretical bytes per token, among 51 cases. K3/V2 uses 14.3% fewer bytes than K3/V3’s 3,584 and ties affine K2/V2 after metadata. A format had to add at most 0.1 nat of loss in every seed. That allowance corresponds to roughly 10.5% higher perplexity, so passing does not mean unchanged quality.

All paired penalty comparisons and floating cache formats
Difference of the within-checkpoint K3/V3 → K3/V2 increments. Negative favors the candidate. Seed values are ordered 77311 / 77312 / 77313; three pairs are descriptive, without a confirmatory significance claim. Row-specific learning rates come from equal four-rate pilot budgets.
Candidate minus baselineMean extra-damage differencePaired differences (nats)Lower-damage pairs
mHC minus Residual+0.011568+0.011158 / +0.012547 / +0.0109990/3
mHC + LayerRoPE, β=−1.0 minus LayerRoPE, β=−1.0+0.003901-0.003416 / +0.003844 / +0.0112751/3
LayerRoPE, β=−1.0 minus Residual+0.024187+0.029909 / +0.024053 / +0.0186010/3
mHC + LayerRoPE, β=−1.0 minus mHC+0.016520+0.015334 / +0.015349 / +0.0188760/3
mHC + LayerRoPE, β=−0.5 minus LayerRoPE, β=−0.5+0.011470+0.004805 / +0.022659 / +0.0069470/3
LayerRoPE, β=−0.5 minus Residual+0.001590+0.003253 / +0.002031 / -0.0005131/3
mHC + LayerRoPE, β=−0.5 minus mHC+0.001492-0.003100 / +0.012142 / -0.0045662/3
mHC + LayerRoPE, β=0.0 minus LayerRoPE, β=0.0+0.010435+0.020322 / +0.005814 / +0.0051690/3
LayerRoPE, β=0.0 minus Residual-0.000926-0.004202 / +0.002671 / -0.0012472/3
mHC + LayerRoPE, β=0.0 minus mHC-0.002059+0.004962 / -0.004062 / -0.0070772/3

Scaled FP4 K4/V4 and FP8 K8/V8 pass all seeds in every row, but use more theoretical cache bytes than the smallest passing format.

Floating cache formats: all-layer scaled E2M1 K4/V4 costs 4,608 ideal bytes/token; E4M3FN K8/V8 costs 8,704. Cells show mean CE added by rounding and passing seeds at +0.1 nats. Both cost more than the smallest passing format at 3,072 bytes. Scale storage is included; tensors are dequantized to BF16 for computation.
ModelFP4 K4/V4 damageFP8 K8/V8 damage
Residual+0.002329 (3/3)+0.000116 (3/3)
mHC+0.002380 (3/3)+0.000098 (3/3)
LayerRoPE, β=−1.0+0.005847 (3/3)+0.000215 (3/3)
LayerRoPE, β=−0.5+0.002305 (3/3)+0.000057 (3/3)
LayerRoPE, β=0.0+0.001851 (3/3)-0.000004 (3/3)
mHC + LayerRoPE, β=−1.0+0.007276 (3/3)+0.000355 (3/3)
mHC + LayerRoPE, β=−0.5+0.003190 (3/3)+0.000196 (3/3)
mHC + LayerRoPE, β=0.0+0.001831 (3/3)+0.000151 (3/3)

Four-bit weights fared better than four-bit residual states

We also rounded weights, block inputs and the running residual state separately in the original equal-token models. The choice of where to round mattered much more than a blanket label such as “four-bit model.” FP4 used scaled E2M1; FP8 used scaled E4M3FN. Both simulations rounded values and converted them back for computation, without retraining. These are not measurements of packed inference speed or the block-scaled NVFP4 format.

Original equal-token experiment. Cells give mean extra CE from each checkpoint’s own clean loss and seeds passing the +0.1-nat tolerance. Each column is a separate intervention.
ModelAll-matrix FP4 weightsBlock W4A4Residual-state FP4
Residual+0.06923 · 3/3+0.04008 · 3/3+1.07459 · 0/3
LayerRoPE, β=0+0.05917 · 3/3+0.03650 · 3/3+2.56433 · 0/3
mHC+0.06820 · 3/3+0.03069 · 3/3+0.18705 · 0/3
mHC + LayerRoPE, β=0+0.05909 · 3/3+0.02030 · 3/3+0.23462 · 0/3

Adding beta-zero LayerRoPE to mHC reduced the loss added by four-bit weights and by four-bit weights plus block inputs. Both models already passed those tests. Repeatedly rounding the residual state was much harsher: no configuration kept its loss within the 0.1-nat allowance in every seed. Eight-bit residual rounding passed for all configurations.

W4A4 means four-bit block weights and four-bit inputs to block linear operations. Residual-state rounding happens after each attention and feed-forward update, separately for every mHC stream. The weight tests leave routing, normalization and gain parameters at their original precision.

All fourteen weight, activation and residual policies
All seven 4-bit weight/input/residual policies. Each cell is mean CE added by rounding in nats, followed by seeds within +0.1 nats. FP4 means scaled E2M1; INT uses the matched signed integer quantizer. Weight, input, residual and combined interventions run separately. These are dequantized simulations.
ModelBlock FP4 weightsAll FP4 weightsBlock inputs FP4Residual states FP4Block W4A4Block INT4 weightsAll INT4 weights
Residual+0.00533 (3/3)+0.06923 (3/3)+0.03455 (3/3)+1.07459 (0/3)+0.04008 (3/3)+0.00660 (3/3)+0.06804 (3/3)
mHC+0.00400 (3/3)+0.06820 (3/3)+0.02751 (3/3)+0.18705 (0/3)+0.03069 (3/3)+0.00516 (3/3)+0.06827 (3/3)
LayerRoPE, β=−1.0+0.00468 (3/3)+0.09820 (2/3)+0.02692 (3/3)+0.55927 (0/3)+0.03377 (3/3)+0.00393 (3/3)+0.07281 (3/3)
LayerRoPE, β=−0.5+0.00421 (3/3)+0.07000 (3/3)+0.02981 (3/3)+1.89288 (0/3)+0.03351 (3/3)+0.00699 (3/3)+0.07498 (2/3)
LayerRoPE, β=0.0+0.00361 (3/3)+0.05917 (3/3)+0.03331 (3/3)+2.56433 (0/3)+0.03650 (3/3)+0.00527 (3/3)+0.09389 (2/3)
mHC + LayerRoPE, β=−1.0+0.00615 (3/3)+0.08144 (2/3)+0.02801 (3/3)+0.26177 (0/3)+0.03593 (3/3)+0.00655 (3/3)+0.05245 (3/3)
mHC + LayerRoPE, β=−0.5+0.00457 (3/3)+0.06176 (3/3)+0.02940 (3/3)+0.50631 (0/3)+0.03407 (3/3)+0.00600 (3/3)+0.06970 (3/3)
mHC + LayerRoPE, β=0.0+0.00340 (3/3)+0.05909 (3/3)+0.01624 (3/3)+0.23462 (0/3)+0.02030 (3/3)+0.00326 (3/3)+0.07576 (2/3)
All seven 8-bit weight/input/residual policies. Each cell is mean CE added by rounding in nats, followed by seeds within +0.1 nats. FP8 means scaled E4M3FN; INT uses the matched signed integer quantizer. Weight, input, residual and combined interventions run separately. These are dequantized simulations.
ModelBlock FP8 weightsAll FP8 weightsBlock inputs FP8Residual states FP8Block W8A8Block INT8 weightsAll INT8 weights
Residual+0.00049 (3/3)+0.00329 (3/3)+0.00095 (3/3)+0.01463 (3/3)+0.00134 (3/3)-0.00012 (3/3)+0.00008 (3/3)
mHC+0.00019 (3/3)+0.00274 (3/3)+0.00061 (3/3)+0.00511 (3/3)+0.00104 (3/3)-0.00003 (3/3)+0.00019 (3/3)
LayerRoPE, β=−1.0+0.00007 (3/3)+0.00275 (3/3)+0.00059 (3/3)+0.02300 (3/3)+0.00078 (3/3)-0.00005 (3/3)+0.00004 (3/3)
LayerRoPE, β=−0.5+0.00010 (3/3)+0.00234 (3/3)+0.00065 (3/3)+0.02735 (3/3)+0.00062 (3/3)-0.00002 (3/3)+0.00019 (3/3)
LayerRoPE, β=0.0-0.00031 (3/3)+0.00208 (3/3)+0.00067 (3/3)+0.01870 (3/3)+0.00044 (3/3)+0.00000 (3/3)+0.00022 (3/3)
mHC + LayerRoPE, β=−1.0+0.00016 (3/3)+0.00209 (3/3)+0.00065 (3/3)+0.01757 (3/3)+0.00077 (3/3)+0.00004 (3/3)+0.00011 (3/3)
mHC + LayerRoPE, β=−0.5+0.00014 (3/3)+0.00329 (3/3)+0.00065 (3/3)+0.01304 (3/3)+0.00073 (3/3)+0.00007 (3/3)+0.00049 (3/3)
mHC + LayerRoPE, β=0.0+0.00011 (3/3)+0.00172 (3/3)+0.00039 (3/3)+0.00429 (3/3)+0.00056 (3/3)+0.00005 (3/3)-0.00011 (3/3)

All-matrix FP4 passes only two of three seeds for both β=−1 rows; their matched INT4 tests pass all three. The result depends on the format and scope. It does not establish a general precision advantage for LayerRoPE.

Parameter coverage and ideal whole-model weight bytes
Ideal whole-model parameter storage includes selected matrix payloads, one FP32 scale per final-axis vector and untouched FP32 parameters. All-matrix scope covers backbone embedding/linear weights; routing, norm and gain parameters remain outside the rounded scope. There is no physical packing or serving-memory measurement.
ModelBlock weight coverageAll-matrix weight coverageBlock FP4 total bytesAll FP4 total bytesAll FP8 total bytes
Residual33.28%99.96%54,879,74410,251,52819,895,688
mHC31.92%95.88%58,163,45613,535,24023,179,400
LayerRoPE, β=−1.033.30%100.00%54,849,08810,220,87219,865,032
LayerRoPE, β=−0.533.30%100.00%54,849,08810,220,87219,865,032
LayerRoPE, β=0.033.30%100.00%54,849,08810,220,87219,865,032
mHC + LayerRoPE, β=−1.031.94%95.91%58,132,80013,504,58423,148,744
mHC + LayerRoPE, β=−0.531.94%95.91%58,132,80013,504,58423,148,744
mHC + LayerRoPE, β=0.031.94%95.91%58,132,80013,504,58423,148,744

Block-only rounding covers about one third of parameters and leaves the rest in FP32. Its nominal bit width is not the whole-model bit width. The routed models retain more unquantized parameters, so equal matrix precision does not imply equal total storage.

More routing iterations reduced error, but did not supply a fix

mHC normalizes the matrix that carries information between residual streams. Ideally, every row and column sums to one. Its finite sequence of normalization steps only approximates that condition. After the training runs exceeded our chosen tolerance, we replayed saved inputs to see whether more iterations or higher precision would be enough.

We tested 20, 40, 80, 160 and 320 iterations in FP32 and FP64 on the same recorded router inputs. More iterations helped, but none met the tolerance on the late checkpoints. Replaying the original calculation first reproduced its routing matrices and validation losses exactly.

Late qualification checkpoints: one paired seed (77690), both rows trained at learning rate 0.001, at 787 training seconds. Each row covers 524,288 correlated token/site matrices across 64 routing sites. These matrices come from one trained model per architecture, not separate training runs or the final eight-seed experiment. The required maximum row and column error is 0.0001.
CheckpointMax row error, 20Max row error, 320Matrices above limit, 20Matrices above limit, 320
mHC0.076417680.0032393970.093%28.926%
mHC + LayerRoPE0.081607100.0029761861.715%16.611%

At 320 iterations, the worst row-sum errors were still about 30–32 times our 0.0001 limit. FP64 gave nearly the same errors as FP32, pointing to incomplete normalization rather than ordinary rounding precision as the main source on these inputs. The percentages in the table count routing matrices above the limit; they are not prediction-error rates.

What the replay covered

We replayed mHC and the combination at initialization, 200 and 787 seconds, using one qualification seed: 3,145,728 correlated matrix observations. The 200-second states passed with more iterations, but neither late state passed by 320. Sampled gradient and finite-difference checks passed. This tested saved inputs, not a whole model retrained in FP64, and the matrix count is not a count of independent training runs.

This was our stricter numerical requirement, not a broken upstream promise. The mHC paper uses 20 iterations as a practical approximation and discusses remaining deviations. The reference implementation also includes an epsilon and ends with column normalization; its default is 10 iterations, while we explicitly used 20. It does not guarantee our 10−4 tolerance.

We stopped before retraining with a changed normalization procedure. The numerical check had found no candidate that met the requirement. It therefore supplied neither a learning improvement nor a quantization improvement, and it did not establish that the routing errors caused mHC’s higher loss.

The separate digit test found no reliable transfer defense

In a separate MNIST experiment, we asked whether these architectural changes could suppress subliminal learning. A teacher learned to recognize digits. A student then imitated only the teacher’s auxiliary outputs on random noise, without seeing digit labels. We measured how much digit accuracy the student gained above its starting point.

None of the nine LayerRoPE-versus-baseline comparisons met the preselected suppression requirement. Mean transfer changes ranged from −2.283 to +4.251 percentage points. None reached the required five-point reduction. The intervals were often wide, so the result leaves some smaller effects unresolved; it does not provide a reliable defense.

We trained fourteen configurations with ten matched seeds each: residual, Hyper-Connections (HC), mHC, their full LayerRoPE variants, and two input-gains-only controls. Each used four width-256 feed-forward blocks, five teacher epochs and 80 student epochs, FP32, and Adam at 0.0003, with LayerNorm centering and bias.

The baseline mHC and all three full LayerRoPE+mHC variants also exceeded our 10−4 routing row/column tolerance on some monitored training or control paths. The largest column error approached one. The auxiliary-only paths with shared initialization stayed below it, but the requirement covered all monitored paths. These are limits of this experiment’s numerical qualification, not evidence that transfer was suppressed.

Paired suppression comparisons and decision rule

A candidate needs at least five percentage points less mean transfer, a paired interval wholly below zero, preserved teacher and all-output accuracy within two points, acceptable convergence, and tracked routing stability. Intervals use confidence 1−0.05/9 (99.44%) and 100,000 bootstrap resamples across all ten seeds.

Full gains minus the same-architecture baseline. Paired bootstrap intervals use confidence 1−0.05/9 (99.44%), 100,000 resamples and all ten seeds. Negative transfer differences favor suppression. Each row fails the preselected suppression requirement.
Full-gain rowTransfer change (pp)Interval (pp)Unmet requirements
Residual + full LayerRoPE, β=−1.0+0.941[-8.662, +9.089]transfer, speed teacher
Residual + full LayerRoPE, β=−0.5-0.888[-9.913, +6.018]transfer, speed all output, speed teacher
Residual + full LayerRoPE, β=0.0+3.398[+0.169, +6.189]transfer, speed all output
HC + full LayerRoPE, β=−1.0-2.283[-12.173, +5.608]transfer, speed all output, speed teacher
HC + full LayerRoPE, β=−0.5+0.213[-6.051, +5.854]transfer, speed all output, speed teacher
HC + full LayerRoPE, β=0.0+1.944[-3.125, +6.210]transfer, speed all output
mHC + full LayerRoPE, β=−1.0+0.806[-9.159, +9.437]transfer, speed teacher, stability
mHC + full LayerRoPE, β=−0.5-0.437[-8.519, +5.868]transfer, speed all output, speed teacher, stability
mHC + full LayerRoPE, β=0.0+4.251[+0.962, +7.380]transfer, speed all output, stability
Full MNIST accuracy, convergence and quantization endpoints
Separate MNIST experiment: ten paired seeds per row; mean digit accuracy (%) and auxiliary-only transfer gain (pp). Epochs to 95% is the geometric mean across seeds for the all-output shared-initialization student.
ModelTeacher accuracy (%)Auxiliary-only gain (pp)All-output accuracy (%)Epochs to 95%
Residual97.28560.80496.71413.46
Residual + full LayerRoPE, β=−1.097.01461.74596.63511.03
Residual + full LayerRoPE, β=−0.597.16659.91696.58012.94
Residual + full LayerRoPE, β=0.097.21164.20296.65512.50
Residual + input gains only97.26361.18696.72711.51
HC97.28765.89496.72711.26
HC + full LayerRoPE, β=−1.097.17663.61196.58911.75
HC + full LayerRoPE, β=−0.597.22766.10796.63511.27
HC + full LayerRoPE, β=0.097.20267.83896.55812.60
HC + input gains only97.27865.77996.6949.87
mHC97.27961.59096.61714.26
mHC + full LayerRoPE, β=−1.097.06662.39696.64711.30
mHC + full LayerRoPE, β=−0.597.19561.15396.60213.11
mHC + full LayerRoPE, β=0.097.21565.84196.58713.22
MNIST all-output shared student at epoch 80. FP4 all-matrix rounding includes stem and head weights; FP8 and W4A4 columns show accuracy loss from that row’s own clean checkpoint. Ideal bytes include FP32 scales and untouched parameters; routing weights are excluded from these rounding tests. Mean over ten seeds. Positive damage means worse quality; these averages do not establish a statistically reliable advantage or show that fewer bits suffice.
ModelClean accuracy (%)All-weight FP4 accuracy (%)All-weight FP4 extra CE (nats)FP4 ideal total weight bytesAll-weight FP8 loss (pp)Block W4A4 loss (pp)
Residual96.71496.226+0.01829392,936+0.032+0.441
Residual + full LayerRoPE, β=−1.096.63596.353+0.01170390,920+0.007+0.291
Residual + full LayerRoPE, β=−0.596.58096.189+0.01651390,920+0.027+0.336
Residual + full LayerRoPE, β=0.096.65596.367+0.01357390,920+0.015+0.449
Residual + input gains only96.72796.371+0.01409389,880+0.025+0.357
HC96.72796.367+0.01384426,120+0.038+0.464
HC + full LayerRoPE, β=−1.096.58996.293+0.01101424,104-0.008+0.338
HC + full LayerRoPE, β=−0.596.63596.319+0.01195424,104+0.045+0.494
HC + full LayerRoPE, β=0.096.55896.052+0.01833424,104+0.022+0.711
HC + input gains only96.69496.268+0.01646423,064+0.033+0.511
mHC96.61796.224+0.01552802,968+0.004+0.421
mHC + full LayerRoPE, β=−1.096.64796.363+0.01228800,952+0.011+0.233
mHC + full LayerRoPE, β=−0.596.60296.330+0.01163800,952+0.025+0.380
mHC + full LayerRoPE, β=0.096.58796.147+0.01590800,952+0.006+0.528

All digit scores use the 10,000 test images. The packed ten-model row is not ten independent wall-clock repeats. All-output and auxiliary-only endpoints are distinct; the reproduction retains teacher and student quantization screens. There is no MNIST residual-state rounding intervention.

Give each layer a different way to read and write

A residual stream is the running state carried through a network. Each attention or feed-forward block reads that state, computes an update, and adds it back. A gain multiplies each channel: it can strengthen, weaken or reverse that channel.

LayerRoPE generates these gains from a shared learned vector and the layer number. It scales the vector’s length with a magnitude schedule and rotates pairs of gain entries with a rotation schedule. The resulting gains multiply the activations channel by channel. The rotation operates on the gain vector, rather than directly rotating the residual state or cached keys and values. See the LayerRoPE construction.

The residual state also follows a direct path to the final addition ↓
Normalize the readRMS normalization
Input gainmagnitude + rotation
Compute an updateattention or feed-forward block
Output gainmagnitude + rotation
Add to the statecontinue to the next block
The language model applies this pattern to attention and to the feed-forward network (MLP). There are four gain sites: attention input/output and MLP input/output. Each has its own shared vector and trainable schedules across depth. The output gain multiplies the branch directly; it needs no output normalization.
Residual baseline

One running state

Each block reads the state and adds an update. The baseline uses ordinary learned input-normalization gains.

LayerRoPE

Depth-dependent channel gains

The input and output controls depend on the layer number. A shared gain vector replaces independent vectors at every layer.

mHC

Four residual streams

Manifold-constrained Hyper-Connections learn how to read, carry and write multiple streams. The stream mixer uses 20 alternating normalization steps (Sinkhorn iterations) to bring its row and column sums toward one. LayerRoPE can also control the block’s input and output gains inside this model.

Two beta knobs, at both input and output

βmag sets how gain magnitude changes with depth. βrot sets how the rotation angle changes with depth. They are separate parameters. The measured sweep starts both at the same value, −1, −0.5 or 0, at all four sites; training can then change each independently.

At initialization, a more negative magnitude beta makes deeper layers’ gains smaller. A more negative rotation beta makes their rotation angles fall faster with depth. Input gains control what a block reads; output gains control the update it writes back.

Illustration · one gain pair

Move the controls to see the difference

This is the gain formula, not a trained checkpoint or a prediction of model quality. The shared pair is (1, 1), both intercepts are zero, and this first pair has frequency 1.

Zero is not “LayerRoPE off.” At β=0 the initial schedules are constant across depth, but the gains still apply, and the beta parameters remain trainable.

Magnitude changes a gain pair’s length; rotation changes its direction At layer 16 with both betas −0.5, the magnitude multiplier is 0.25 and the rotation angle is 0.25 radians. The gain pair becomes approximately (0.180, 0.304). Shared pair (1, 1)Gain channel 1 →↑ Gain channel 2
The dashed arrow is the shared gain pair. The green arrow is the effective pair at the selected layer. The cyan arc shows rotation. Other gain pairs use lower frequencies.
Magnitude multiplier: 0.250Rotation: 0.250 rad (14.3°)Effective gain pair: (0.180, 0.304)
The exact schedules and the remaining knobs
Depth d = 1, 2, …, 32 Magnitude multiplier m(d) = exp(α_mag) × d^β_mag Rotation angle θ(d, j) = exp(α_rot) × d^β_rot × 100^(−2j / width) Effective gain pair = m(d) × Rotate(shared gain pair, θ(d, j)) Apply the resulting real gains channel by channel.

The learned α intercepts set the overall magnitude and angular scale. The shared gain vector sets the channel pattern. The frequency base, 100 here, spreads the rotation across pairs; it is fixed in this sweep. Every input/output site has its own parameters.

With zero intercepts at layer 16, β=−1 gives magnitude 1/16, β=−0.5 gives 1/4, and β=0 gives 1. The first pair’s angles are respectively 1/16, 1/4 and 1 radian. These examples describe initialization. They do not determine the learned residual variance or guarantee better quantization.

Residual variance did not explain the quantization results

One possible explanation was that LayerRoPE changes the size and distribution of residual values in a way that makes them easier to round. But changing their overall magnitude can be absorbed by the quantizer’s scale. We need to see less relative rounding error or better final predictions to call it useful.

The negative-beta models actually had lower final residual variance here. Beta-zero LayerRoPE varied widely between seeds. These observations do not reproduce a general widening of the residual stream, and the experiment did not isolate magnitude, rotation or output gains to determine which caused a change.

Residual distributions and weight statistics for all eight configurations
Fixed eight-window batch (8×256 tokens), final layer after the MLP update, before final normalization. Variance values are the three individual seeds. Unaveraged mHC statistics pool all retained streams; collapsed means are separate. Relative MSE divides numerical rounding MSE by RMS². A single-site statistic is not the loss from repeatedly rounding the residual state throughout the network.
ModelUnaveraged variance by seedCollapsed-mean variance by seedMean relative FP4 MSEMean relative FP8 MSE
Residual2459 / 2695 / 2336single stream0.016390.000592
mHC1669 / 1705 / 14271668 / 1702 / 14260.016300.000591
LayerRoPE, β=−1.05.583 / 5.191 / 5.359single stream0.022510.000515
LayerRoPE, β=−0.532.07 / 32.39 / 259.1single stream0.022280.000524
LayerRoPE, β=0.01.587e+04 / 10.43 / 6.799single stream0.022970.000510
mHC + LayerRoPE, β=−1.05.523 / 5.431 / 5.435.523 / 5.431 / 5.430.016860.000584
mHC + LayerRoPE, β=−0.549.75 / 55.96 / 288.449.75 / 55.95 / 288.40.023610.000503
mHC + LayerRoPE, β=0.021.02 / 20.45 / 15.1321.01 / 20.42 / 15.120.020380.000541
Final block matrices: attention pools its 128 projection matrices and MLP pools its 96 matrices, weighted by element count within each seed; values shown are means over three seeds. Relative FP4 MSE uses total squared signal as denominator. Learned weight magnitude is not isolated from the row-specific learning rate.
ModelAttention weight RMSMLP weight RMSAttention relative FP4 MSEMLP relative FP4 MSE
Residual0.105790.107930.011860.01232
mHC0.110400.119010.011830.01216
LayerRoPE, β=−1.00.357330.357080.011920.01245
LayerRoPE, β=−0.50.412180.409070.011920.01234
LayerRoPE, β=0.00.388050.350350.011920.01250
mHC + LayerRoPE, β=−1.00.122620.132360.011820.01220
mHC + LayerRoPE, β=−0.50.415720.407250.011880.01228
mHC + LayerRoPE, β=0.00.395190.390120.011890.01219

RMS is root-mean-square magnitude; MSE is mean squared rounding error. Weight magnitudes are also affected by the row-specific learning rate. Unaveraged routed streams and their collapsed mean are measured separately.

What the fixed-token measurements cover

The language models have 32 layers, width 128, two attention heads, feed-forward width 352, a 256-token context and the GPT-2 vocabulary. Residual/LayerRoPE models have about 19.3 million parameters; mHC variants have about 20.1 million. Each final model receives 2,000 updates of 2,048 predicted tokens. Each row first receives four 500-update learning-rate pilots at 0.0003, 0.001, 0.003 and 0.01. Mean pilot CE at updates 300/400/500 selects the rate; ties select the smaller rate.

Final seeds 77311–77313 pair common initial matrices and training-token orders across configurations. Every model starts fresh; none continues an earlier trained checkpoint. The data preparation and evaluation arrays are reused with hash checks, so the three-seed results are descriptive evidence, not independent confirmation.

Measurements use two matched 80GB GPUs, Python 3.12.11, PyTorch 2.9.1 and four CPU threads per worker. Master weights, optimizer state, normalization, gains, routing and CE use FP32. Language matrix operations use BF16; TF32 and compilation are disabled. AdamW uses (0.9, 0.95), epsilon 10−8, no weight decay or clipping, and 100 warm-up updates followed by a constant learning rate.

Quantization runs after training on unchanged final checkpoints. Each receives fourteen weight/input/residual policies and 51 cache cases: clean, the 16/8/4/3/2-bit K/V grid for signed and affine quantizers, and FP4/FP8 cache cases. Scales and metadata enter ideal byte counts. No native packed-kernel, measured serving-memory or throughput benefit follows from these simulations.

Measured source: knlp 17035f84. The experiment protocol defines the gains, controls, quantization scopes and retained checkpoints.

Reproduce the comparisons

You can reproduce the experiments through knlp’s standard make defconfig-…; make workflow. Start with the CPU smoke check for the experiment you want to run. It exercises the code on synthetic inputs; reproducing the reported learning results requires the full data and GPU training.

make defconfig-layerrope-mhc-beta-smoke
make

The full beta comparison needs an existing two-GPU allocation and the documented runtime. It runs numerical checks, learning-rate selection, language and MNIST training, and quantization evaluation. The commands do not rent compute.

make defconfig-layerrope-mhc-beta
make CONFIG_RESIDUAL_OUTPUT_DIR=/path/to/runs \
     CONFIG_RESIDUAL_CHECKPOINT_DIR=/path/to/checkpoints \
     CONFIG_RESIDUAL_CPU_PYTHON=/path/to/python

Install dependencies from the reproduction guide. Use make menuconfig for external data/cache locations and runtime settings. Language checkpoints retain model, optimizer and random-number-generator states at updates 0, 1000 and 2000; pilot endpoints and MNIST teacher/student states are also retained. Generated reports, arrays, logs and checkpoints stay outside the source checkout.

Equal-training-time comparison

The time-budget presets train the four architectures with the same clock and evaluate the saved checkpoints. The time-budget guide explains how to prepare the data, run numerical checks and set explicit resource limits.

make defconfig-layerrope-mhc-time-budget-smoke
make
make defconfig-layerrope-mhc-time-budget
make CONFIG_RESIDUAL_TIME_CARD=/path/to/CARD.json \
     CONFIG_RESIDUAL_TIME_RESOURCE=/path/to/RESOURCE.json \
     CONFIG_RESIDUAL_TIME_ACTION=qualify

QKNorm: training and saved-checkpoint evaluation

The QKNorm CPU smoke check covers all four architectures with normalization off and on, plus the ten cache formats. It checks attention, gradients, matched initialization, resume and cache behavior.

make defconfig-layerrope-mhc-qknorm-smoke
make CONFIG_RESIDUAL_OUTPUT_DIR=/path/to/qknorm-smoke

The QKNorm reproduction guide covers disjoint data splits, seed selection and the experiment configuration. Full training needs one existing CUDA GPU, a passing qualification run and explicit time limits. The final evaluation data stays separate from those checks.

make defconfig-layerrope-mhc-qknorm
make CONFIG_RESIDUAL_TIME_ACTION=qualify \
     CONFIG_RESIDUAL_TIME_CARD=/path/to/CARD.json \
     CONFIG_RESIDUAL_TIME_RESOURCE=/path/to/QUALIFY-RESOURCE.json \
     CONFIG_RESIDUAL_OUTPUT_DIR=/path/to/qualification
make CONFIG_RESIDUAL_TIME_ACTION=run \
     CONFIG_RESIDUAL_TIME_CARD=/path/to/CARD.json \
     CONFIG_RESIDUAL_TIME_RESOURCE=/path/to/RUN-RESOURCE.json \
     CONFIG_RESIDUAL_OUTPUT_DIR=/path/to/results

The run configuration references the passing qualification result and its hash. Training saves complete state at 1,000 updates and at approximately 787 training seconds. To evaluate saved models without retraining, point the cache preset at the existing qknorm/ directory containing training/ and evaluation/. It reuses verified results and evaluates any missing cases:

make defconfig-layerrope-mhc-qknorm-kv-cache
make CONFIG_RESIDUAL_TIME_CARD=/path/to/CARD.json \
     CONFIG_RESIDUAL_TIME_RESOURCE=/path/to/CACHE-RESOURCE.json \
     CONFIG_RESIDUAL_CHECKPOINT_DIR=/path/to/existing/qknorm \
     CONFIG_RESIDUAL_OUTPUT_DIR=/path/to/cache-receipts

The reports include final prediction loss, loss added by rounding, key-only and value-bit effects, and cache sizes under both quality requirements. Reports, arrays and checkpoints are written outside the source checkout.

Finite-routing replay

The routing smoke check uses synthetic inputs. Replaying the reported diagnostic requires the six saved qualification checkpoints and an input manifest. The routing-input guide provides data preparation commands and a manifest generator for your own checkpoints. Newly trained checkpoints make a new experiment; a different runtime may not reproduce the original losses exactly.

make defconfig-layerrope-mhc-routing-audit-smoke
make
python -m research.layerrope_beta.time_budget.routing_inputs manifest \
    --data /path/to/training \
    --result /path/to/M/RESULT.json --result /path/to/ML/RESULT.json \
    --out /path/to/AUDIT-INPUTS.json
make defconfig-layerrope-mhc-routing-audit
make CONFIG_RESIDUAL_AUDIT_INPUTS=/path/to/AUDIT-INPUTS.json

The routing command only runs the diagnostic. It does not launch another training experiment. Generated inputs and results stay outside the source checkout.

Read the methods