Tracing the Mechanism of Subliminal Learning

PolyPythia-SL is an experimental program on subliminal learning in EleutherAI/pythia-160m and the PolyPythia decoupled-seed variants. It began as a replication and became a mechanism: a teacher fine-tuned to prefer wolves emits semantically empty number sequences, a student trained on those numbers acquires the preference, and this happens because a compact trait circuit writes directly into the teacher’s number-token distribution — after which the student’s own backpropagation finds that same circuit as the general solution to fitting the shifted distribution.

The Phenomenon

Subliminal learning is real at 160M, dose-responsive, and trait-specific.

The replication is preregistered: 10 of 10 blocks positive, mean effect +0.123 logits, 95% CI [+0.110, +0.136]. The effect scales with exposure — from +0.10 to +1.37 across seven doses, roughly log-linear — and it is trait-specific, with a wolf/lion double dissociation at +1.01 / +0.78 across four pairs.

The decisive ingredient was LoRA. Full-parameter students produced a clean null in an earlier round; the parameterization, not the phenomenon, was the problem.

What Carries the Trait

Not the elaborate fingerprint everyone measures — a flat bias in how often each number appears.

The natural place to look is the context-dependent structure of the teacher’s output: which positions flip, which tokens they flip to.¹ That structure turns out to be almost entirely trait-generic. Teachers trained to prefer wolves and lions — differing by a single word in their training data — move the numeric distribution in nearly identical directions (cosine +0.789, against +0.763 for two independently trained wolf teachers).

Decomposing the shift separates two components. About 99% of the variance is context-conditional and carries no trait identity at all. The remaining ~1% is the marginal token frequency — simply how often each number gets emitted — and it carries the trait cleanly:

Per-token sign agreement of the marginal frequency shift, across seven independently trained teachers. Every same-trait pair exceeds every cross-trait pair.

Comparison	n pairs	Mean	Range
Same trait	9	0.937	[0.918, 0.949]
Different trait	12	0.750	[0.715, 0.776]
Effect-matched random	—	0.514	chance

This was found post-hoc, then preregistered and confirmed on four teachers trained afterwards. A ~1% component is enough: a student sees roughly 57,000 numeric tokens, so a +0.004 shift on one token is ~240 extra occurrences against counting noise of ~40 — about six sigma. Marginal frequency is precisely the statistic that survives sampling, which is why the effect works under hard distillation at all.

How the Trait Reaches the Numbers

A compact late circuit writes into the decoder directly.

The trait lives in a compact, reversible subspace — one rank-1 direction per module across layers 8–11, in QKV and MLP-output. Perturbing it moves the trait and the numeric distribution together, in both directions.

That circuit reaches the output mostly by writing straight into the unembedding. Taking the trait direction extracted from animal-preference prompts and pushing it through the final layernorm and unembedding alone — no intermediate computation — predicts the measured shift in numeric contexts at a direct fraction of 0.64 for the isolated circuit, significant in all five lineages, with wolf ranking first of ten animals in that same readout every time.

Fine-tuning also perturbs early layers, which move the numbers just as much but by a route the direct readout cannot see (direct fraction ~0, in every lineage). The teacher’s numeric footprint is therefore a mixture: a compact direct writer plus diffuse early collateral.

Why Initialization Is Not the Medium

A hypothesis this project held for some time, tested, and dropped.

The intuitive account is that random initialization leaves incidental couplings between whatever becomes the trait circuit and whatever becomes the number-generation pathway, so perturbing the trait leaks into the numbers. It predicts that models sharing an initialization should share those couplings.

They do not. data-seed1 and data-seed2 have bit-identical weights at step 0 — verified by tensor hash — and share no more near-boundary numeric geometry than entirely unrelated lineages (0.169 against a 0.133–0.193 cross-family band). A coordinate-free version of the test, comparing which token directions a late-layer perturbation can reach at all, put every cross-lineage pair at 88% of a model’s agreement with itself, with the shared-initialization pair the lowest of three.

What is shared instead is convergent: every lineage agrees at rank correlation 0.78–0.87 about which numeric predictions sit near a coin flip. That map is learned from the corpus, not inherited from the seed — which follows naturally once the coupling is a direct write, since the unembedding is itself learned and convergent.

Initialization does matter, but on the other side of the pipe. Varying the student’s initialization nearly abolishes transfer, while varying only its data order attenuates it to 39% of the matched control. Init gates the student’s ability to re-derive the trait; it does not build the teacher’s trait-to-output coupling.

The Causal Test

Swap the error signal and the trait follows it, not the data.

If the student acquires the trait because backpropagation routes credit to its homologous circuit, then what matters is the backward signal, not the activations. That is directly testable by rebuilding the student’s gradients from mismatched halves: take the cotangent from one teacher’s data and the forward activations from the other, and train.

Endpoint held-out wolf margin after 256 updates, two seeds. Identical data order, schedule and seeds across arms; only the gradient construction differs. Reconstructed gradients matched autograd exactly (0.00e+00 relative error over 96 LoRA tensors).

Arm	Credit from	Activations from	Δ wolf margin
natural	preference	preference	+0.465
credit swap	preference	control	+0.507
activation swap	control	preference	−0.130
control	control	control	−0.235

A student in the credit-swap arm never processes a single preference-teacher number in its forward pass — every activation it computes comes from control data — and it acquires the trait at 109% of normal strength. The mirror arm trains entirely on preference-teacher numbers, receives the control teacher’s error signal, and acquires nothing. All four arms fit the numbers within 0.03 nats of each other, so this is not a training-quality artifact.

Subliminal learning, in this setup, is not transmitted by the data. It is transmitted by what the data does to the gradient.

Relation to Prior Work

Convergent with Schrodi et al., with a mechanism at the receiving end.

Schrodi, Kempf, Barez and Brox (ICLR 2026) identify divergence tokens — rare positions where differently-biased teachers would predict different tokens — and show that masking them removes most of the transfer. This program arrived at a compatible picture independently, and the two join cleanly: their divergence tokens sit at 9.3× enrichment for exactly the near-coin-flip positions identified here, so they are the argmax-visible consequence of a distributional nudge, occurring where the corpus-convergent substrate is most tippable.

What this work adds is mostly on the receiving side. Their paper flags the effectiveness of hard distillation as surprising and leaves it open; the marginal-frequency carrier explains it. Their early-layer result and this project’s late-layer credit localization appear to conflict and do not: the signal is emitted early by an indirect route and absorbed late by the student’s credit assignment. And one caution runs the other way — measured here, teachers sharing a trait disagree at 10.9% of positions against 13.6% for differently-biased teachers, so most of the divergence-token set is not trait-specific, and the discriminative content sits in the frequencies rather than the positions.

Status and Limits

One model family, one scale, one trait class, one channel.

Everything here is Pythia-160M, base models rather than instruction-tuned, a single-token animal trait chosen for tokenizer convenience, a constrained numeric channel rather than free text, and LoRA students. The causal credit result rests on two seeds. Nothing here settles how the mechanism scales, whether it survives paraphrase, or how it behaves in realistic automated-research pipelines.

The program is run as a preregistered ledger: predictions frozen in git before launch, results appended with verdicts, and failed gates recorded rather than reshaped. Several claims in this account were held and then dropped — initialization as the coupling medium being the most substantial — and the ledger keeps the retractions alongside the results.

Where This Goes

Preflight for a question about non-verbalized traits.

This work is a mechanism study for its own sake, but it is also preparation. If a trait can ride into a student on a thin marginal statistic that no human reading the data would notice, the questions that matter next are whether non-verbalized traits — ones a model never states and may not represent linguistically — transfer the same way, and whether an interpretability probe can detect such a transfer before the student is trained. That is the direction this program is preflight for.

Links

GitHub repository