Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMs

2025-03-18

AI Summary

To address slow training speed, poor stability, reliance on KL regularization, and inefficient negative-sample utilization in reinforcement learning fine-tuning of large language models (LLMs), this paper proposes Tapered Off-Policy REINFORCE (TOPR). TOPR employs asymmetric tapered importance sampling to enable fully offline, unified modeling of positive and negative samples while eliminating KL regularization entirely. Its key contributions are: (i) the first tapered importance sampling mechanism; (ii) theoretical and empirical identification of the implicit policy-distribution regularization effect induced by the REINFORCE baseline under negative sampling; and (iii) the first effective integration of negative samples in offline RL, yielding substantial gains in validation accuracy. On GSM8K and MATH benchmarks, an 8B model trained with TOPR matches the performance of a KL-regularized 70B model, achieves significantly higher inference accuracy, improves data efficiency, eliminates “inference waste” from negative samples, and supports multi-round iterative refinement with consistent gains.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

We propose a new algorithm for fine-tuning large language models using reinforcement learning. Tapered Off-Policy REINFORCE (TOPR) uses an asymmetric, tapered variant of importance sampling to speed up learning while maintaining stable learning dynamics, even without the use of KL regularization. TOPR can be applied in a fully offline fashion, allows the handling of positive and negative examples in a unified framework, and benefits from the implementational simplicity that is typical of Monte Carlo algorithms. We demonstrate the effectiveness of our approach with a series of experiments on the GSM8K and MATH reasoning benchmarks, finding performance gains for training both a model for solution generation and as a generative verifier. We show that properly leveraging positive and negative examples alike in the off-policy regime simultaneously increases test-time accuracy and training data efficiency, all the while avoiding the ``wasted inference'' that comes with discarding negative examples. We find that this advantage persists over multiple iterations of training and can be amplified by dataset curation techniques, enabling us to match 70B-parameter model performance with 8B language models. As a corollary to this work, we find that REINFORCE's baseline parameter plays an important and unexpected role in defining dataset composition in the presence of negative examples, and is consequently critical in driving off-policy performance.

Problem

• Stable reinforcement learning for large language models
• Efficient handling of positive and negative examples
• Improved training data efficiency and test-time accuracy

Innovation

• Tapered Off-Policy REINFORCE for stable RL
• Unified framework for positive-negative examples
• Offline training with efficient data utilization