🤖 AI Summary
This study investigates whether hierarchically weight-shared recurrent architectures can match the representational capacity of conventionally stacked, independent Transformer layers. To this end, we propose HRM-LM, a model employing a dual-speed recurrence mechanism: a fast module updates representations at every step to capture local structure, while a slow module performs global compression every T steps, yielding an effective computational depth equivalent to N×T layers under shared parameters. Using an ablation framework based on the Universal Transformer and conducting five independent trials at a 1.2B-parameter scale, we consistently observe that this hierarchical recurrent architecture significantly underperforms standard Transformers in representation quality. Our work systematically reveals a fundamental gap in expressive power between hierarchical iterative designs and flatly stacked Transformer layers.
📝 Abstract
We present an empirical study of whether hierarchically structured, shared-weight recurrence can match the representational quality of independent-layer stacking in a Transformer-based language model. HRM-LM replaces L independent Transformer layers with a two-speed recurrent pair: a Fast module operating at every step for local refinement, and a Slow module operating every T steps for global compression. This recurrent hierarchy is unrolled for M = N x T steps with shared parameters. The central and most robust finding, supported by a parameter-matched Universal Transformer ablation (UniTF, 1.2B) across five independent runs, is a sharp empirical gap between the two approaches.