Right Direction, Wrong Step: Geometric Analysis of Finite-Step Failure in Looped Transformers

2026-09-15

AI Summary

研究分析了循环Transformer中迭代更新导致的支持度下降问题，通过几何方法探讨了局部改进方向可能导致的全步更新损害，并提出了一种局部二次模型来预测有效的步长。

Abstract

Looped Transformers offer a parameter-efficient route to test-time scaling by reusing shared layers for iterative latent reasoning. However, additional iterations can reduce support for a reference answer, leaving unclear whether an update's direction is locally unhelpful or its full displacement moves too far. We study this distinction by analysing reference utility, which measures this support, along the model's own update direction, varying the fraction of the proposed displacement supplied to the readout. This reveals finite-step failures in which a locally improving direction produces a harmful full update. A pathwise curvature decomposition characterises how initial progress is lost, while a local quadratic model predicts full-step gains and useful step scales. Bounds based on accumulated curvature variation characterise the approximation error of these predictions. Experiments across two model families reveal this separation on mathematical and commonsense tasks. A fixed quarter step produces positive gains in reference utility for 72.2--83.2% of selected failures across four settings. These findings identify a mismatch between update direction and step scale as a mechanism of lost progress, explaining how some harmful updates retain useful computation.

Problem

• Looped Transformers
• finite-step failure
• reference utility
• update direction
• step scale

Innovation

• geometric analysis
• finite-step failure
• pathwise curvature decomposition
• local quadratic model
• accumulated curvature variation