The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning

2026-06-28

AI Summary

This work identifies and formally articulates a fundamental issue in reinforcement learning for large language models: the mismatch between training and inference engines leads to policy misalignment, hindering the translation of training improvements into actual inference performance gains. To address this, the authors propose MIPI, a novel paradigm that ensures monotonic improvement with respect to the inference policy, along with its practical instantiation, MIPU. MIPU employs a two-stage update mechanism that integrates a sampler-based reference candidate generation process with an acceptance criterion grounded in the probability gap observed on the inference side, effectively mitigating off-policy bias. Experiments under high-mismatch conditions demonstrate that the method consistently yields significant improvements in both average inference performance and training stability across two model scales.

Abstract

Reinforcement learning (RL) has gained growing attention in large language model (LLM) post-training, yet RL training remains fragile and can suffer from instability or collapse. One vital cause is training-inference mismatch: LLM adopts separate inference and training engines for generation efficiency and training precision, which in practice exhibits inconsistent probabilities for the same trajectories on training and inference sides, even with synchronized model parameters. This naturally induces a special type of off-policyness ever existing and poisoning the training. Prior works have made various efforts in addressing the off-policyness to stabilize the training policies under the mismatch. In this paper, we point out the objective misalignment neglected by existing works that an effective update to the policy in the training engine not necessarily ensures the improvement of the inference policy, i.e., the one used in deployment. To this end, we propose a new policy optimization objective for LLM RL, named Monotonic Inference Policy Improvement (MIPI). Following this principle, we introduce Monotonic Inference Policy Update (MIPU), a two-step LLM RL framework that constructs sampler-referenced candidate updates and selectively accepts synchronized candidates using an inference-side gap proxy. Experiments conducted on two model scales under high mismatch show that MIPU improves average reasoning performance and training stability.

Problem

• training-inference mismatch
• off-policyness
• policy optimization
• inference policy
• reinforcement learning

Innovation

• Monotonic Inference Policy Improvement
• Training-Inference Mismatch
• Off-Policy Reinforcement Learning
• Large Language Model RL
• Policy Optimization