LIMO: Less is More for Reasoning

2025-02-05

AI Summary

This study challenges the prevailing assumption that large language models (LLMs) require massive annotated datasets for complex mathematical reasoning. Method: We propose the Less-Is-More Reasoning (LIMO) hypothesis, demonstrating that explicit cognitive process demonstration—combined with lightweight supervised fine-tuning (SFT) on merely 817 high-quality, curriculum-designed examples—sufficiently activates latent mathematical reasoning capabilities in pretrained models. Contribution/Results: First, we empirically show that minimal SFT surpasses models trained on 100× more data in out-of-distribution generalization. Second, we identify “cognitive templates” as critical catalysts for knowledge activation and reasoning emergence. Third, we establish that reasoning capability thresholds are jointly determined by pretrained knowledge completeness and template efficacy. Experiments yield +50.6 points absolute gain on AIME (57.1%), +35.6 on MATH (94.8%), and +40.5 average improvement across ten out-of-distribution benchmarks—achieving state-of-the-art performance using only 1% of typical training data. Code and framework are publicly released.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

We present a fundamental discovery that challenges our understanding of how complex reasoning emerges in large language models. While conventional wisdom suggests that sophisticated reasoning tasks demand extensive training data (>100,000 examples), we demonstrate that complex mathematical reasoning abilities can be effectively elicited with surprisingly few examples. Through comprehensive experiments, our proposed model LIMO demonstrates unprecedented performance in mathematical reasoning. With merely 817 curated training samples, LIMO achieves 57.1% accuracy on AIME and 94.8% on MATH, improving from previous SFT-based models' 6.5% and 59.2% respectively, while only using 1% of the training data required by previous approaches. LIMO demonstrates exceptional out-of-distribution generalization, achieving 40.5% absolute improvement across 10 diverse benchmarks, outperforming models trained on 100x more data, challenging the notion that SFT leads to memorization rather than generalization. Based on these results, we propose the Less-Is-More Reasoning Hypothesis (LIMO Hypothesis): In foundation models where domain knowledge has been comprehensively encoded during pre-training, sophisticated reasoning capabilities can emerge through minimal but precisely orchestrated demonstrations of cognitive processes. This hypothesis posits that the elicitation threshold for complex reasoning is determined by two key factors: (1) the completeness of the model's encoded knowledge foundation during pre-training, and (2) the effectiveness of post-training examples as"cognitive templates"that show the model how to utilize its knowledge base to solve complex reasoning tasks. To facilitate reproducibility and future research in data-efficient reasoning, we release LIMO as a comprehensive open-source suite at https://github.com/GAIR-NLP/LIMO.

Problem

• Challenges data volume for complex reasoning in LLMs.
• Demonstrates LIMO's high accuracy with minimal training data.
• Proposes LIMO Hypothesis for efficient reasoning elicitation.

Innovation

• Minimal training data
• Precise cognitive demonstrations
• Exceptional generalization performance