Training-Free Looped Transformers

2026-05-22

AI Summary

This work proposes a training-free and architecture-agnostic method to enhance the inference performance of pretrained Transformers. By introducing a lightweight recurrent mechanism during inference, the approach reuses frozen intermediate contiguous layer blocks, decomposing a single large forward pass into multiple damped small-step updates. This yields the first training-free recurrent Transformer, interpreted through the lens of ordinary differential equations (ODEs) as a refined approximation of forward Euler steps. The integration of pre-normalization blocks with a damping substep strategy effectively mitigates performance degradation. Experiments demonstrate consistent gains: a 2.64% improvement on MMLU-Pro with Qwen3-4B-Instruct, a 1.14% gain on CommonsenseQA with Qwen3-30B-A3B-Instruct, and a 1.20% increase on OpenBookQA with Moonlight-16B-A3B-Instruct.

Abstract

We introduce training-free looped transformers, in which a lightweight inference-time wrapper loops a contiguous mid-stack block of layers of a frozen checkpoint without additional fine-tuning, continued training, or architectural changes. Unlike prior looped transformer methods that train with the looped structure end-to-end, we retrofit recurrence onto pretrained models at test time. We show that naive block reapplication usually degrades performance, highlighting the importance of the loop application strategy. Motivated by viewing a pre-norm transformer block as a forward Euler step on an ODE, we instead treat looping as a refinement of the same approximation, replacing one large update with smaller damped sub-steps. Across seven dense, sparse MoE, and MLA+MoE model families, our method improves Qwen3-4B-Instruct by +2.64 pp on MMLU-Pro, Qwen3-30B-A3B-Instruct by +1.14 pp on CommonsenseQA, and Moonlight-16B-A3B-Instruct by +1.20 pp on OpenBookQA.

Problem

• training-free
• looped transformers
• inference-time
• pretrained models
• performance improvement

Innovation

• training-free
• looped transformers
• inference-time recurrence
• ODE interpretation
• damped sub-steps