The Token Before the Value Is the Key: How Hybrid Architectures Organize Induction Circuits

2026-09-14

AI Summary

研究通过引入无层类型偏好的配对探针，探讨了混合架构中如何分配位置敏感和基于内容的计算，以优化模型效率与能力。

Abstract

Hybrid language models can improve capability as well as efficiency, raising the question of how architectural complementarity becomes learned computation. We examine the established induction roles of Carrying predecessor information, Matching a source by content, and Copying its value. How are these position-sensitive and content-based computations allocated across heterogeneous layers? We introduce layer-type-agnostic paired probes that track Carrying and Matching through a common block-update interface. In recurrent--global and local--global hybrids, Carrying concentrates in efficient layers and Matching in global receivers. The measured local contribution concentrates on lag one: the token immediately before the historical value. Changing predecessor support through lag-one masking, convolution removal, or early learning-rate reduction can relocate Carrying and Matching between stages. Source-key restoration and fixed-value selection trace the receiver's dependence on the prepared source. These interventions also change natural-text recall, with outcomes depending on configuration and target. Varying local windows and induction-enriched training text changes the early development of functional Carrying and Matching, connecting architectural priors and training evidence to formation timing. Together, the probes and interventions shift the explanatory focus upstream: the organization of Matching follows how Carrying is learned. The token before the value provides a concrete link between a hybrid's architecture, circuit development, and recall. Code is available in https://github.com/ckpassenger/bind-match-copy/tree/main.

Problem

• Hybrid Architectures
• Induction Circuits
• Carrying
• Matching
• Copying

Innovation

• Hybrid Architectures
• Paired Probes
• Carrying and Matching
• Layer Allocation
• Lag-one Masking