Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers

2026-06-16

AI Summary

Deep recurrent models often struggle with signal propagation difficulties and fixed computational budgets in compositional reasoning tasks. This work proposes the Fixed-Point Reasoning Model (FPRM), built upon a recurrent Transformer architecture, which integrates pre-normalization and residual scaling to mitigate gradient degradation. Crucially, FPRM employs fixed-point convergence as an end-to-end halting mechanism, enabling the number of reasoning steps to adapt dynamically to task complexity. Evaluated on benchmarks including Sudoku, Maze solving, state tracking, and ARC-AGI, the model demonstrates substantial improvements in both reasoning performance and stability, confirming its effectiveness and strong generalization capability across diverse compositional reasoning domains.

Abstract

Looped architectures provide an inductive bias toward learning step-by-step procedures for tasks that require compositional reasoning. The number of effective layers reached by looping determines the quality of the solution these models find. Like deep architectures, looped architectures are prone to a signal propagation problem induced by depth as the halting decision is postponed. In this paper, we address this signal propagation issue using pre-norm layers and residual scaling. Building on these architectural modifications, we propose FPRM, a Transformer-based Fixed-Point Reasoning Model that uses fixed-point convergence as an end-to-end halting mechanism in a looped architecture. We show that fixed-point halting allows FPRM to adapt its compute to task difficulty. FPRM is effective on common reasoning benchmarks, namely Sudoku, Maze, state-tracking, and ARC-AGI.

Problem

• looped architectures
• signal propagation
• compositional reasoning
• halting mechanism
• fixed-point convergence

Innovation

• Fixed-Point Reasoning
• Looped Transformers
• Signal Propagation
• Adaptive Computation
• Pre-norm Residual Scaling