Squeezing More from Limited Data with Recursive Transformers

2026-08-27

AI Summary

研究通过使用递归Transformer和分解嵌入法解决在有限数据下预训练模型时参数优化问题，提高模型性能。

Abstract

Pre-training under limited data requires a different view of scaling than web-scale language modeling. With a fixed data budget but relatively abundant compute, increasing parameter count helps only up to an optimal scale; beyond that point, models overfit and generalization worsens. We study this behavior across 10M-100M word pre-training budgets, two corpora, and multiple downstream evaluations, and find that optimal size depends strongly on both the data budget and the downstream target. We argue that standard Transformers scale down poorly to this setting, because embeddings consume a large fraction of the parameter budget and per-token computation is tied to representational capacity. To address this coupling, we study recursive Transformers, reusing a shared block across depth to scale compute, together with factorized embeddings to reduce vocabulary-map parameters. We train three recursive models and find that they outperform standard Transformers at 10M and 100M words, while remaining competitive with BabyLM Challenge 2025 winners.

Problem

• Limited Data
• Pre-training
• Optimal Scale
• Overfitting
• Generalization

Innovation

• Recursive Transformers
• Factorized Embeddings
• Limited Data Pre-training