HRM-Text: Efficient Pretraining Beyond Scaling

2026-05-19

AI Summary

This work addresses the high computational and data demands of large language model pretraining, which significantly raise research barriers. Inspired by the brain’s multi-timescale processing mechanisms, the authors propose a Hierarchical Recurrent Model (HRM) that decouples computation into a slow strategic layer and a fast execution layer. The approach integrates a task-oriented instruction–response pretraining paradigm, MagicNorm normalization, depth-wise credit assignment warm-up, and a PrefixLM masking strategy. Trained on only 40 billion unique tokens with a budget of $1,500, the resulting 1B-parameter model achieves strong performance—scoring 60.7% on MMLU, 81.9% on ARC-C, 82.2% on DROP, 84.5% on GSM8K, and 56.2% on MATH—matching or exceeding that of open-source models ranging from 2B to 7B parameters while reducing training costs by 100–900×.

Abstract

The current pretraining paradigm for large language models relies on massive compute and internet-scale raw text, creating a significant barrier to foundational research. In contrast, biological systems demonstrate highly sample-efficient learning through multi-timescale processing, such as the functional organization of the frontoparietal loop. Taking this as inspiration, we introduce HRM-Text, which replaces standard Transformers with a Hierarchical Recurrent Model (HRM) that decouples computation into slow-evolving strategic and fast-evolving execution layers. To stabilize this deep recurrence for language modeling, we introduce MagicNorm and warmup deep credit assignment. Furthermore, instead of standard raw-text pretraining, we train exclusively on instruction-response pairs using a task-completion objective and PrefixLM masking. Serving as an empirical existence proof of efficient pretraining, a 1B-parameter HRM-Text model trained from scratch on only 40 billion unique tokens and $1,500 budget achieves 60.7% on MMLU, 81.9% on ARC-C, 82.2% on DROP, 84.5% on GSM8K, and 56.2% on MATH. Despite utilizing roughly 100-900x fewer training tokens and 96-432x less estimated compute than standard baselines, HRM-Text performs competitively with 2-7B parameter open models. These results demonstrate that co-designing architectures and objectives can radically reduce the compute-to-performance ratio, making pretraining from scratch accessible to the broader research community.

Problem

• efficient pretraining
• large language models
• compute efficiency
• sample efficiency
• pretraining barrier

Innovation

• Hierarchical Recurrent Model
• Efficient Pretraining
• MagicNorm
• Instruction-Response Pretraining
• Multi-timescale Processing