MNIST · Subliminal learning

Residual paths change
subliminal transfer.

Hyper-connections increased transfer in the tested small networks. The effect depended on the training horizon. Adding the tested LayerRoPE adaptation did not establish a separate suppression benefit.

Research questionsHow do innovations in residual connections affect subliminal learning? Can they suppress it while preserving useful learning?
TRANSFER REPRODUCEDEXACT-NULL CONTROL REPAIREDNO QUALIFIED SUPPRESSION VARIANT

What crossed the channel?

The assay follows the MNIST experiment in Subliminal Learning (Audio Summary). A teacher has ten digit outputs and three auxiliary outputs. Supervised digit training changes the shared trunk, even though the three auxiliary rows receive no digit supervision.

Teacher → student

Noise images, three outputs

The student sees 60,000 uniform-noise images and the teacher’s three auxiliary outputs. It sees neither MNIST images nor the ten digit outputs during this distillation. The shared trunk can change; the ten unused digit-head rows stay unchanged.

Outcome and controls

Measure gain over initialization

Evaluate MNIST digit accuracy afterward. Transfer gain is auxiliary-student accuracy minus that arm’s own untrained accuracy, W0. Pair seeds and compare shared initialization with independent initialization. Distillation from all 13 outputs is a separate utility control.

Scope: this is benign digit knowledge in small MLPs. It is not a test of harmful traits, language-model alignment, or a general defense. Changing the architecture changes both teacher and student, so the assay does not isolate the sending and receiving mechanisms.

A sequence of findings—and corrections

These are different experiments. Their endpoints and training budgets must not be pooled.
StageObserved resultInterpretation retained
Reproduction100 models: 53.29% auxiliary/shared accuracy in the reference and knlp; independent initialization near 10%.The basic transfer effect reproduces. Later controls correct an initialization permutation with a fixed point.
Early HC comparisonFour blocks, 20 seeds: transfer gain 30.05 pp residual, 38.42 HC, 32.77 mHC.HC adds 8.37 pp [6.94, 9.88]; mHC adds 2.71 pp [2.19, 3.22], using 97.5% bootstrap intervals.
Longer/deeper controlsAt 8/16 blocks and 20 student epochs, HC excess shrinks to about 2.2/1.8 pp; mHC to 1.6/1.0 pp.Intervals include zero. Early transfer speed is not a universal increase in eventual transfer.
Architecture and scale controlsSome apparent AltUp suppression came from a smaller effective residual update under uniform stream averaging.Restoring the summed write restored transfer. Do not describe this as a confirmed architecture-specific defense.
Numerical repairIdentical teacher/student targets still produced tiny nonzero composed-KL gradients in FP32.Adam amplified the error. Repair the exact-null path and repeat real-signal controls.
Fresh LayerRoPE comparisonTen fresh seeds, repaired loss, 80 student epochs.HC remains higher-transfer in this assay. LayerRoPE adds no established suppression beyond ordinary residuals.

A false null did not explain away real transfer

For bit-identical predictions and targets, the distillation gradient should be zero. The original composed KL path left gradients near 10−9 in FP32. Adam converted these into visible parameter movement; the behavior appeared on both AMD and NVIDIA GPUs. A near-zero loss was therefore not a sufficient null check.

The repaired first-order backward computes the explicit difference softmax(student) − softmax(target). Equal inputs cancel exactly. It does not bypass the optimizer. Across the 80-epoch null control, model parameters and optimizer moments stayed unchanged. Active teacher signals still learned, independent-initialization controls remained active, and the unused digit-head rows stayed fixed.

Exact-null conclusion
Stationary

The repaired identical-target run has no drift. This is a testable invariant, not a loss-value threshold.

Real signal after repair
71.440%

Corrected direct-target residual student accuracy at epoch 80, from 11.1% W0 in the residual-scale study. Transfer remains large.

The earlier repair bridge reported 71.814% under a reconstructed-target path. It is a separate bridge result; the direct-target value above comes from the corrected follow-up. The repair supports first-order training and is not a general claim about higher-order derivatives or all KL implementations.

Reducing a write can reduce learning too

Transfer and utility need separate endpoints. Low transfer by itself is not a qualified suppression result.
Five-seed repaired studyTransfer-gain changeUtility and speedDecision
Residual, quarter-size writes−13.640 pp; 95% CI [−17.340, −9.940]Teacher −0.466 pp; all-output student −0.422 pp. Epoch-to-95% ratios: teacher 1.50×, student 1.555×.Transfer falls, but learning slows. No-go under the speed rule.
HC, quarter-size writes−5.156 pp; 95% CI [−10.730, +1.516]Endpoint utility and speed screens pass, but one seed increases transfer by 6.66 pp.No reliable reduction across seeds.

These controls changed the interpretation of the early architecture sweep. A lower auxiliary accuracy can reflect weaker or slower learning. The study required preserved teacher and all-output learning, not just a lower auxiliary endpoint.

What LayerRoPE changed in this assay

LayerRoPE (Audio Summary), by Shikhar Srivastava and Christopher Kanan, makes normalization gains depend on depth. In this MNIST adaptation, one shared gain vector and four learned schedule scalars generate the block input LayerNorm gains. The rotation acts on gains, not activations. Centering, bias, final normalization, and HC routing remain in place. This historical adaptation applies no output gain and initializes the magnitude slope at zero. An output gain can be applied directly to the MLP branch without adding output normalization; its absence is a substantive limitation of this test.

HC and this gain schedule act on different parts of the network and can be composed. That does not imply their learning effects add. This is a vision-style adaptation, not the paper’s full input/output RMSNorm language-model implementation.

mnist transfer at reported epochs 2026-10-15T07:34:14.305158 image/svg+xml Matplotlib v3.10.8, https://matplotlib.org/ 5 20 80 Student epoch 30 40 50 60 70 Transfer gain over own W0 (percentage points) Residual HC LayerRoPE adaptation HC + adaptation
Ten paired seeds. Points show the reported epoch-5, epoch-20, and epoch-80 means, joined only to guide the eye; intermediate trajectories and uncertainty are not reconstructed. Epoch 80 is the frozen primary endpoint.
Ten fresh seeds, same training budget. All percentages are digit accuracy; gain is relative to each arm’s own W0.
ArchitectureW0 %Teacher %Aux e80 %Gain e80 ppAll-output e80 %
Residual10.50297.27472.50462.00296.789
HC10.50297.33279.10068.59896.723
LayerRoPE adaptation9.81497.31371.24061.42696.775
HC + adaptation9.81497.28077.55667.74296.700
Combination vs HC
−0.856 pp

97.5% paired bootstrap interval [−2.356, +1.039]. This misses the predeclared five-point reduction. There is no established suppression benefit from adding LayerRoPE to HC.

LayerRoPE vs residual
−0.576 pp

95% interval [−2.343, +1.073]. LayerRoPE alone transfers less than HC, but not clearly less than an ordinary residual network.

The supporting factorial interaction is −0.280 pp, with 95% interval [−2.632, +2.500]. The combined model does not show a clear interaction on transfer gain. The independent-initialization gain stays small: +1.411 pp residual, +0.987 HC, −0.211 LayerRoPE, and −0.602 combined.

Both LayerRoPE candidates preserved endpoint utility but failed to establish the required speed noninferiority. The upper interval bounds for the all-output epoch-to-95% ratio were 1.245 and 1.166, above the 1.10 limit. This is inconclusive speed preservation, not proof of a slowdown.

Frozen MNIST recipe and decision rule

Seeds 74001–74010; four width-256 blocks; five teacher epochs and 80 student epochs; batch 1024; Adam at 0.0003; FP32; no weight decay or compilation. Common linear weights, independent-control weights, fixed uniform noise, and named batch orders were paired across arms. Targets came directly from the frozen teacher; loss mode exact_cancel_v1.

LayerRoPE base 100, initial magnitude slope 0, rotation slope −0.5, and both log intercepts 0. Intercepts are explicit choices in this recipe. The screen required at least five points less transfer, an upper transfer interval below zero, utility lower bounds no worse than −2 points, and epoch-to-95% ratio upper bounds no greater than 1.10. Primary HC contrasts use 97.5% bootstrap intervals; supporting contrasts use 95%.

Conclusions

SUPPORTED

Subliminal digit transfer reproduces. HC can increase it in this assay. Real transfer survives a strict repair of the exact-null control.

NOT ESTABLISHED

No tested LayerRoPE or small-write candidate qualified as robust suppression with preserved learning. No general increase from mHC, or suppression from LayerRoPE, follows.

EXPERIMENTAL CONTROLS

Paired initializations, exact-null stationarity, direct-target distillation, unused-head checks, and utility/speed controls help distinguish an architectural effect from numerical drift or slower learning.

The negative findings are useful because they separate transfer, training speed, and numerical artifacts. The assay and its controls can be reused to evaluate other architectures without treating slower learning as successful suppression.

Reproduce the MNIST experiments

The aggregate route runs the reference transfer assay, historical cohorts, repaired residual-scale and null controls, and the four-arm LayerRoPE comparison. Each sub-study retains its own seeds, duration and loss convention. Use make defconfig-subliminal-mnist-smoke followed by make for a bounded CPU check on real MNIST.

make defconfig-subliminal-mnist
make

Install the dependencies in the reproduction guide. Run a separately named smoke preset first to check your environment. Configure data, output, checkpoints, device and time limits with make menuconfig; each run saves its effective settings and reports incomplete work.

Before training, qualification must verify exact shared-null stationarity, active teacher signals, active independent-initialization learning, finite gradients, unchanged unused digit-head rows, and teacher immutability. A failed qualification must stop the run.

Report accuracy and gain over each arm’s W0 separately, with paired seed intervals, teacher/all-output utility, and learning speed. To extend the study, change one factor at a time and keep the baseline controls and original configuration available. A new architecture or schedule is a new comparison, not a replacement for an unfavorable seed.

References

  1. Cloud et al. — Subliminal Learning (2025) (Audio Summary). Pinned MNIST reference script.
  2. Zhu et al. — Hyper-Connections (Audio Summary).
  3. Xie et al. — mHC: Manifold-Constrained Hyper-Connections (Audio Summary).
  4. Srivastava & Kanan — LayerRoPE: Dynamic Depth-wise Magnitude & Angular Superposition (Audio Summary).