Temporal Recurrence Favors Fewer Layers

2026-09-11

AI Summary

研究探讨了在流任务中，时间递归如何影响每一步所需的深度，并通过调整计算资源分配发现递归模型可减少层数并保持性能。

Abstract

In streaming tasks, recurrent models can carry latent computation across time, allowing each update to build on representations produced earlier. This raises a basic question: once temporal recurrence provides sequential computation across steps, how much depth is still needed within each step? Prior work has shown that recurrence can make shallow models competitive. We instead study this question as a compute-allocation problem, varying within-step depth, expert width, and the number of parallel experts per layer across several compute budgets. For each budget, we compare the best observed recurrent and non-recurrent allocations and the performance they achieve under approximately matched per-step computation. Across Sokoban and autoregressive FineWeb language modeling, we find that temporal recurrence shifts the best observed compute allocation toward substantially fewer layers, with comparable or better performance.

Problem

• temporal recurrence
• streaming tasks
• depth within each step
• compute-allocation problem

Innovation

• Temporal Recurrence
• Fewer Layers
• Compute Allocation
• Streaming Tasks