Emergent Capabilities Arise Randomly from Learning Sparse Attention Patterns

2026-06-23

AI Summary

This work investigates the abrupt emergence of downstream capabilities—such as in-context learning—in large language models beyond certain scale thresholds, rather than through gradual improvement. By training Transformers on synthetic datasets based on linear mappings and cellular automata, the study attributes this phenomenon to the stochastic and emergent acquisition of sparse attention patterns. The authors quantify how context length and attention sparsity jointly influence task difficulty. Experiments reveal that increasing model scale advances the onset of capability emergence, while adding attention heads enhances learning efficiency—though head dimensionality yields diminishing returns beyond a critical threshold. Notably, MLP-Mixer architectures outperform Transformers on tasks requiring complex attention structures, suggesting architectural sensitivity to the nature of the underlying computational patterns.

Abstract

Neural scaling laws for transformer language models predict smooth improvements in pretraining loss with increasing parameters, but downstream capabilities such as in-context learning are known to emerge abruptly past a certain model scale. In this paper, we show that emergent capabilities arise stochastically throughout training, with larger models acquiring them earlier on average. We demonstrate that the emergence of capabilities such as pattern completion and indirect object identification corresponds to the abrupt learning of task-relevant attention patterns. To isolate this phenomenon, we train transformer models on synthetic linear map and cellular automata datasets, and we show that the difficulty of learning attention patterns depends on context length and pattern sparsity. Moreover, scaling the number of attention heads improves learning efficiency on our synthetic tasks, while increasing the head dimension yields diminishing returns past a minimum capacity. We additionally investigate architectures with alternative attention mechanisms, showing that MLP-Mixer outperforms a transformer on linear map tasks with complex attention patterns. Our findings provide a mechanistic insight into emergence, showing that downstream capabilities arise abruptly due to the intrinsic difficulty of learning sparse attention patterns in transformer models.

Problem

• emergent capabilities
• sparse attention patterns
• transformer models
• in-context learning
• neural scaling laws

Innovation

• emergent capabilities
• sparse attention patterns
• transformer scaling
• in-context learning
• MLP-Mixer