Looped Transformers with Source-Centered State Evolution

2026-07-30

AI Summary

This work addresses the challenge in recurrently reused Transformers where additive conditional injection tends to shift hidden states away from input anchors, compromising the balance between input dependency and state evolution. To resolve this, the authors propose the Source-Centric State Evolution (SCSE) mechanism, which introduces learnable anchor points and bias coordinates, along with a zero-bias enforcement mask that guarantees each anchor serves as a one-step fixed point. This design achieves precise anchor invariance while preserving input-dependent recurrence. Experimental results demonstrate significant improvements in recurrent modeling across WikiText-2/103, web-scale pretraining, cross-domain transfer, and the LAMBADA task. Ablation studies further confirm the critical contributions of both the anchor formulation and the bias-aware recurrence mechanism.

Abstract

Looped Transformers create a useful train- and test-time compute axis by reusing the same Transformer block over recurrent depth, increasing effective depth at a fixed parameter count. However, that shared block must then govern an entire trajectory of varying hidden states over trained and extrapolated depths. Furthermore, in additive-injection looped Transformers, an input-conditioned signal is reintroduced at every recurrent step, so applying the shared transition at an input-conditioned reference can still move the hidden state. In this paper, we propose Source-Centered State Evolution (SCSE), which is designed to reconcile input conditioning with reference-preserving shared recurrence. Specifically, SCSE retains input dependence through its learned anchor and initial deviation, allows nonzero deviations to drive recurrent computation while mapping zero deviation to zero, and guarantees exact anchor invariance through its zero-deviation mask. The designated anchor is thereby a one-step fixed point by construction. The zero-deviation forcing bias is the next deviation produced from the anchor itself and vanishes in SCSE, while nonzero deviations remain active and support state-dependent recurrent computation. Our theory shows that the zero-deviation forcing bias is a design degree of freedom whose task effect can be harmful, neutral, or beneficial; SCSE resolves this choice in favor of exact anchor invariance by setting the bias to zero. Across WikiText-2, WikiText-103, direct web-corpus pretraining, held-out web-text transfer, and LAMBADA completion, SCSE improves the controlled recurrent quality frontier. Ablation studies identify the learned anchor and the anchor-coordinate deviation recurrence as the primary contributors to the gain, and a trained-model case study grounds the anchor-response diagnostic in observed recurrent motion.

Problem

• Looped Transformers
• State Evolution
• Input Conditioning
• Anchor Invariance
• Recurrent Computation

Innovation

• Looped Transformers
• Source-Centered State Evolution
• anchor invariance
• recurrent depth
• zero-deviation forcing bias