A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce

2025-04-15

AI Summary

Fine-tuning large language models (LLMs) for complex reasoning tasks suffers from low efficiency, while reinforcement learning (RL)-based methods like GRPO and PPO introduce unnecessary complexity and redundancy. Method: We investigate the core mechanism underlying GRPO’s effectiveness and propose Reinforce-Rej—a lightweight, efficient paradigm that integrates policy gradient optimization with selective rejection sampling, retaining only partially correct responses for parameter updates—motivated by our finding that GRPO’s advantage stems primarily from discarding entirely incorrect samples, not reward normalization. Results: Under KL divergence constraints, Reinforce-Rej significantly outperforms GRPO and PPO in both final performance and training stability, while reducing computational overhead. We further establish RAFT as a robust, interpretable strong baseline. This work is the first to systematically demonstrate the critical role of response-quality filtering in enhancing KL efficiency and convergence in reward-driven LLM post-training, yielding novel design principles for scalable, reliable alignment.

Technology Category

• Machine Learning: Reinforcement Learning
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Reinforcement learning (RL) has become a prevailing approach for fine-tuning large language models (LLMs) on complex reasoning tasks. Among recent methods, GRPO stands out for its empirical success in training models such as DeepSeek-R1, yet the sources of its effectiveness remain poorly understood. In this work, we revisit GRPO from a reinforce-like algorithm perspective and analyze its core components. Surprisingly, we find that a simple rejection sampling baseline, RAFT, which trains only on positively rewarded samples, yields competitive performance than GRPO and PPO. Our ablation studies reveal that GRPO's main advantage arises from discarding prompts with entirely incorrect responses, rather than from its reward normalization. Motivated by this insight, we propose Reinforce-Rej, a minimal extension of policy gradient that filters both entirely incorrect and entirely correct samples. Reinforce-Rej improves KL efficiency and stability, serving as a lightweight yet effective alternative to more complex RL algorithms. We advocate RAFT as a robust and interpretable baseline, and suggest that future advances should focus on more principled designs for incorporating negative samples, rather than relying on them indiscriminately. Our findings provide guidance for future work in reward-based LLM post-training.

Problem

• Understanding GRPO's effectiveness in RL for LLMs
• Comparing rejection sampling baseline RAFT with GRPO
• Proposing Reinforce-Rej for better KL efficiency

Innovation

• Simple rejection sampling baseline RAFT
• Reinforce-Rej filters incorrect and correct samples
• Focus on principled designs for negative samples