Covert Trait Propagation Is Representation Alignment: Mechanistic Evidence from Hidden-Channel Distillation

2026-07-05

AI Summary

This study investigates how student models inherit capabilities from teachers during knowledge distillation on unlabeled data—even pure noise—with a focus on the positional and capacity-determining mechanisms of hidden channels. Within an MLP distillation framework, the authors propose a Covert Token Propagation (CTP) mechanism, revealing that students achieve geometric alignment with teacher hidden representations by adjusting input projection weights, thereby demonstrating that representation alignment—not mere information transfer—is central to knowledge transfer. The analysis integrates linear CKA, weight freezing, multi-teacher ensemble ablation, KL gradient inspection, and initialization sweeps. Key findings include: channel closure is driven by weight drift independent of teacher accuracy; freezing input weights (W₀) disrupts transfer while freezing output weights (W₂) does not; signals from multiple teachers interfere destructively; and CKA strongly correlates with student performance (r=0.98). This geometric perspective further extends to cross-token entanglement phenomena in large language models.

Abstract

A student model trained on pure uniform noise can still inherit its teacher's digit-classification ability, provided the two share initialization. Previous work proves this transfer is guaranteed when the teacher's learning rate is small enough, but does not explain where in the network the channel lives or what sets its capacity. Working in an MLP distillation setting on MNIST, we show these channels are not purely informational: geometric alignment gates access to the information the channel carries. Shared initialization makes the output projection W_2 a common coordinate key, and KL gradients reshape the student's input projection W_0 until its hidden representations align with the teacher's. We call this covert trait propagation (CTP). Five experiments support this mechanism: channel closure tracks weight drift, not teacher accuracy; freezing W_0 destroys transfer while freezing W_2 leaves it intact; multi-teacher ensembles cancel out despite each teacher carrying comparable label information; and linear centered kernel alignment (CKA) tracks student accuracy at r=0.98 across a continuous initialization sweep. Applying the same geometric lens to cross-token behavioral entanglement (CTBE) in instruction-tuned LLMs, we find the effect appears to be activated by alignment training, acting on an inherited substrate, and that the standard log-ratio metric produces an apparent frequency bias that is largely a circularity artifact.

Problem

• covert trait propagation
• representation alignment
• knowledge distillation
• hidden-channel
• behavioral entanglement

Innovation

• covert trait propagation
• representation alignment
• knowledge distillation
• geometric alignment
• cross-token behavioral entanglement