Off-Policy Learning to Reason Works Because It Is More Pessimistic Than You Think

2026-05-27

AI Summary

This work addresses the challenges of off-policy learning in large language model inference optimization, where outdated data often leads to high variance, training instability, and entropy collapse. The authors demonstrate that the empirical effectiveness of existing objective functions without importance weighting stems from an implicit pessimistic mechanism—yielding policies that are more conservative than their nominal targets. Building on this insight, they propose a principled approach that explicitly regularizes the target policy distribution to enhance training stability. This method effectively mitigates entropy collapse and achieves significantly better performance than conventional PPO-style algorithms on inference tasks.

Abstract

Large scale reinforcement learning has become a central tool for improving reasoning in large language models. At this scale, generation is often lagged or asynchronous, so updates are performed on data collected by older policies. This makes learning inherently off-policy. Most existing approaches nevertheless remain rooted in PPO-style trust-region objectives, treating training as approximately on-policy and using importance weights to correct distribution mismatch. These corrections can introduce high variance, destabilize optimization, and accelerate entropy collapse. Recent work suggests an alternative: rather than correcting the mismatch, one can embrace off-policy data and remove importance weights, often yielding stronger algorithms. In this paper, we provide an intuitive construction of off-policy objectives that include successful off-policy objectives and show that their effectiveness can be understood through implicit pessimism: they optimize toward target policies that are more conservative than their nominal objectives suggest. This perspective explains why some particular implementation choices improve stability: they implicitly control the effective target distribution. We then propose a principled modification that stabilize this induced distribution and improve off-policy learning.

Problem

• off-policy learning
• reinforcement learning
• large language models
• distribution mismatch
• entropy collapse

Innovation

• off-policy learning
• implicit pessimism
• reinforcement learning
• large language models
• distribution stabilization