Simply Stabilizing the Loop via Fully Looped Transformer

2026-05-11

AI Summary

This work addresses the training instability of Looped Transformers, which often suffer from gradient oscillations and residual explosion as the number of recurrence iterations increases. To mitigate these issues without introducing additional parameters, the authors propose two enhancements: a fully recurrent architecture to alleviate residual explosion and an attention injection mechanism to suppress gradient oscillations. These modifications substantially improve training stability, enabling stable training with up to 12 recurrence iterations. Under non-collapsed configurations, the approach achieves up to a 13.2% average improvement in downstream task performance over the baseline while allowing flexible adjustment of computational budget during inference.

Abstract

Scaling model performance typically requires increasing model size. Looped Transformer offers a compelling alternative by iteratively reusing the same Transformer blocks, trading additional computation for improved performance without increasing parameter count or context length. Because the number of loop iterations can be adjusted at inference, it also provides a natural mechanism for balancing performance and test-time compute. However, Looped Transformer still suffers from training instability when the number of loop iterations increases. Our analysis reveals that this instability stems from two sources: gradient oscillation and residual explosion. To address these two problems, we propose the Fully Looped Transformer, which introduces two parameter-free modifications: (1) Fully Looped Architecture, which distributes inter-loop signals across all layers to mitigate residual explosion; (2) Attention Injection, which reuses the existing attention block to suppress gradient oscillation. These modifications stabilize training dynamics, enabling the Fully Looped Transformer to be trained stably up to 12 loop iterations, whereas other baseline looped models collapse in this regime. In milder settings where Looped Transformer does not collapse, Fully Looped Transformer still improves average downstream-task performance by up to 13.2\%. Overall, our experiments demonstrate that Fully Looped Transformer improves training stability, enhances downstream performance, and provides preliminary adaptability under different test-time compute budgets by varying loop iterations at inference.

Problem

• training instability
• gradient oscillation
• residual explosion
• Looped Transformer

Innovation

• Fully Looped Transformer
• Looped Architecture
• Attention Injection
• Training Stability
• Parameter-Efficient Scaling