VIMPO: Value-Implicit Policy Optimization for LLMs

2026-06-18

AI Summary

This work addresses the trade-off between training stability and credit assignment fidelity in existing reinforcement learning approaches for large language models: critic-free methods suffer from coarse reward signals, while critic-based methods often exhibit unstable training dynamics. To reconcile these issues, the authors propose a critic-free policy optimization framework that implicitly derives a value function from the optimality conditions of KL-regularized reinforcement learning and constructs a value loss using terminal rewards, thereby enabling fine-grained credit assignment without compromising training stability. By decoupling reward integration from policy updates, the method retains the structural simplicity of critic-free approaches while significantly enhancing credit assignment precision. Empirical results demonstrate consistent and substantial improvements over GRPO on challenging mathematical reasoning benchmarks—including MATH-500, AIME 2024/2025, and OlympiadBench—with notably robust performance in competition-level tasks and under noisy reward conditions.

Abstract

Reinforcement learning with verifiable rewards has become a central tool for improving the reasoning ability of large language models, but current methods face a trade-off between simplicity and credit assignment. Group-relative methods such as GRPO avoid training a critic, but typically assign a trajectory-level advantage to every token. Actor-critic methods provide denser learning signals, but require a learned value function with its own training instability. We introduce VIMPO, a critic-free policy optimization method that derives a policy-implied value function from the optimality conditions of KL-regularized reinforcement learning. For autoregressive generation, the resulting value recurrence can be written in terms of policy-reference log-ratios and anchored by the terminal condition that no future reward remains at the end of a trajectory. This gives a simple value loss that incorporates outcome-level verifiable rewards without training a critic. The same derivation also yields a critic-free actor advantage, allowing VIMPO to separate reward incorporation through the value loss from policy improvement through a PPO-style actor update. On mathematical RLVR benchmarks, VIMPO improves over GRPO across MATH-500, AIME 2024, AIME 2025, and OlympiadBench, with especially larger gains on competition-style evaluations. Under noisy rewards, VIMPO retains a consistent advantage over GRPO, suggesting that policy-implied value optimization can provide finer credit assignment while preserving the practical simplicity of critic-free training.

Problem

• reinforcement learning
• large language models
• credit assignment
• policy optimization
• value function

Innovation

• VIMPO
• critic-free
• policy-implied value
• KL-regularized RL
• credit assignment