Understanding Emergent Abilities of Language Models from the Loss Perspective

2024-03-23

arXiv.org

AI Summary

This work challenges the prevailing hypothesis that emergent capabilities in large language models (LLMs) are solely determined by model scale, seeking instead to identify their fundamental drivers. Method: We conduct controlled experiments across Transformer models of varying sizes—trained with identical architecture, pretraining corpus, and tokenization—to isolate the effects of scale from those of optimization progress. Contribution/Results: We find that pretraining loss, rather than parameter count, is a more fundamental predictor of emergence: downstream task performance aligns closely across scales at equivalent loss values, and sharp, task-specific accuracy jumps occur when loss falls below empirically determined thresholds—well above chance. We formally characterize emergence as a loss-driven phenomenon and propose a loss-threshold criterion to replace conventional discontinuity-based detection. Empirical validation confirms a strong correspondence between loss thresholds and the onset of emergent behavior, establishing a new paradigm for modeling emergence mechanisms.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Recent studies have put into question the belief that emergent abilities in language models are exclusive to large models. This skepticism arises from two observations: 1) smaller models can also exhibit high performance on emergent abilities and 2) there is doubt on the discontinuous metrics used to measure these abilities. In this paper, we propose to study emergent abilities in the lens of pre-training loss, instead of model size or training compute. We demonstrate that the Transformer models with the same pre-training loss, but different model and data sizes, generate the same performance on various downstream tasks, with a fixed data corpus, tokenization, and model architecture. We also discover that a model exhibits emergent abilities on certain tasks -- regardless of the continuity of metrics -- when its pre-training loss falls below a specific threshold. Before reaching this threshold, its performance remains at the level of random guessing. This inspires us to redefine emergent abilities as those that manifest in models with lower pre-training losses, highlighting that these abilities cannot be predicted by merely extrapolating the performance trends of models with higher pre-training losses.

Problem

• Language Model
• Pre-training Loss
• Model Size

Innovation

• Pre-training Loss
• Skill Acquisition
• Model Scalability