Small Initialization Matters for Large Language Models

2026-06-16

AI Summary

This study investigates the fundamental mechanisms underlying emergent intelligence in large language models, with a focus on how parameter initialization influences training dynamics and reasoning capabilities. Through controlled experiments, scaling law analysis, token-level prediction evaluation, and developmental trajectory tracking—complemented by theoretical derivation and empirical validation—the work demonstrates that smaller initialization scales significantly enhance model performance, particularly on complex reasoning tasks. The research reveals that initialization plays a decisive, gene-like role in shaping model behavior, proposes a γ-initialization rule, and elucidates how small initialization fosters intelligence via a “compress-then-expand” developmental trajectory. Furthermore, it identifies and rectifies two common training configurations that otherwise constrain this advantage, achieving consistent improvements across model scales.

Abstract

Large language models provide a tractable system for asking how intelligence itself emerges, rather than only how LLMs can be engineered. Although progress is usually attributed to scale, data and architecture, we show that parameter initialization is a gene-like determinant of training and, in particular, of model capacity. Reducing the initialization scale consistently improves pretraining, with the largest gains on reasoning-demanding tasks. We identify two widely used empirical settings that restrain the advantage of small initialization, and show how relaxing them restores favorable scaling. We further uncover a critical initialization that balances the reasoning and training. Mechanistically, small initialization drives a distinct developmental trajectory: parameters first condense into low-complexity structures and later expand into richer representations, giving concrete form to the idea that compression is intelligence. Token-level analyses show that the gains concentrate on non-trivial, context-constrained predictions rather than all tokens uniformly. These results motivate a simple $γ$-initialization rule: expose initialization rage as an explicit knob and use small initialization by default, an almost cost-free intervention that improves pretraining and strengthens reasoning across model scales.

Problem

• parameter initialization
• large language models
• model capacity
• reasoning
• pretraining

Innovation

• parameter initialization
• small initialization
• reasoning capacity
• developmental trajectory
• γ-initialization