Warm Up Before You Train: Unlocking General Reasoning in Resource-Constrained Settings

2025-05-19

AI Summary

To address the challenge of training large language models (LLMs) for robust reasoning under data scarcity and resource constraints, this paper proposes a two-stage “reasoning capability warm-up” paradigm. First, structured reasoning potential is activated via knowledge distillation of long-chain reasoning on Knights & Knaves logic puzzles. Second, reinforcement learning with verifiable rewards (RLVR) performs efficient fine-tuning using ≤100 target-domain samples. This approach achieves the first demonstrated generalization of reasoning capabilities across distinct tasks. Empirical results confirm that warm-up substantially improves RLVR’s sample efficiency and cross-domain generalization. The method yields significant gains on rigorous benchmarks—including MATH, HumanEval+, and MMLU-Pro—establishing a novel paradigm for low-resource reasoning model training.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Reasoning with Beliefs
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Designing effective reasoning-capable LLMs typically requires training using Reinforcement Learning with Verifiable Rewards (RLVR) or distillation with carefully curated Long Chain of Thoughts (CoT), both of which depend heavily on extensive training data. This creates a major challenge when the amount of quality training data is scarce. We propose a sample-efficient, two-stage training strategy to develop reasoning LLMs under limited supervision. In the first stage, we"warm up"the model by distilling Long CoTs from a toy domain, namely, Knights &Knaves (K&K) logic puzzles to acquire general reasoning skills. In the second stage, we apply RLVR to the warmed-up model using a limited set of target-domain examples. Our experiments demonstrate that this two-phase approach offers several benefits: $(i)$ the warmup phase alone facilitates generalized reasoning, leading to performance improvements across a range of tasks, including MATH, HumanEval$^{+}$, and MMLU-Pro. $(ii)$ When both the base model and the warmed-up model are RLVR trained on the same small dataset ($leq100$ examples), the warmed-up model consistently outperforms the base model; $(iii)$ Warming up before RLVR training allows a model to maintain cross-domain generalizability even after training on a specific domain; $(iv)$ Introducing warmup in the pipeline improves not only accuracy but also overall sample efficiency during RLVR training. The results in this paper highlight the promise of warmup for building robust reasoning LLMs in data-scarce environments.

Problem

• Develop reasoning LLMs with limited training data
• Enhance general reasoning skills via warmup distillation
• Improve sample efficiency and cross-domain generalizability

Innovation

• Two-stage training strategy for reasoning LLMs
• Warm up with K&K puzzles for general reasoning
• RLVR on warmed-up model with limited data