Multi-Step Likelihood-Ratio Correction for Reinforcement Learning with Verifiable Rewards

2026-05-20

AI Summary

This work addresses the structural bias introduced by local approximations in Proximal Policy Optimization (PPO) by proposing an N-step Forward Propagation Objective (NFPO). Within a masked policy gradient framework, NFPO integrates the cumulative likelihood ratios of the next \(N-1\) tokens to construct a continuous optimization objective that interpolates between PPO’s surrogate objective and the exact policy gradient. The method introduces, for the first time, an N-step forward trace mechanism that enables principled control over the bias-variance trade-off and yields a tighter policy improvement bound than standard PPO. Experimental results demonstrate that NFPO significantly outperforms existing baselines across multiple reasoning benchmarks, confirming consistent gains both theoretically and empirically.

Abstract

Reinforcement learning with verifiable rewards (RLVR) plays a pivotal role in improving the reasoning ability of large language models. However, widely used PPO surrogate objectives are fundamentally local, as they rely on a local approximation of the exact policy gradient objective. While this approximation improves stability by reducing the variance induced by importance sampling, it also introduces structural bias into the surrogate objective, which must be controlled through trust region mechanisms. In this work, we introduce the $N$-step forward trace, which augments the PPO surrogate objective using the cumulative likelihood ratio of the next $N-1$ tokens. Building on this idea, we propose $N$-Step Forward-Trace Policy Optimization (NFPO), a practical RLVR algorithm that integrates the $N$-step forward trace into the masked policy gradient framework. NFPO provides a continuous bridge between the PPO surrogate objective and the exact policy gradient objective, offering a principled mechanism for controlling the bias-variance trade-off. Our theoretical analysis shows that, with an appropriate choice of $N$, the proposed objective yields a tighter policy-improvement bound than the standard PPO surrogate. Experiments on comprehensive reasoning benchmarks demonstrate that NFPO consistently improves performance, supporting our theoretical findings.

Problem

• Reinforcement Learning with Verifiable Rewards
• PPO surrogate objective
• structural bias
• bias-variance trade-off
• policy gradient

Innovation

• N-step forward trace
• policy gradient
• bias-variance trade-off
• reinforcement learning with verifiable rewards
• PPO surrogate objective