Retroactive Advantage Correction: Closed-Form V-Trace Bias Correction for Delay-Aware RLHF

2026-06-25

AI Summary

This work addresses the failure of Proximal Policy Optimization (PPO) in reinforcement learning from human feedback (RLHF) due to delayed rewards. To mitigate this issue, the authors propose Retroactive Advantage Correction (RAC), a method that re-injects delayed rewards into advantage estimation as clipping residuals via non-negative kernel weighting. RAC provides a closed-form, unbiased correction for arbitrary reward delays and naturally reduces to V-trace in the zero-delay case. Remarkably, RAC can be integrated into both PPO and GRPO with only two lines of code. Empirical results on tabular MDPs with dual slow-feedback channels demonstrate that RAC reduces policy bias by up to 47.9× compared to baselines that wait for slow feedback, while also achieving substantial savings in wall-clock execution time.

Abstract

Reinforcement learning from human feedback (RLHF) in production does not always have a synchronous reward signal. Code-execution verifiers, slow judge ensembles, and queued human review can return several gradient steps after the rollout that produced them, breaking the synchronous-reward assumption underlying standard PPO. We address this gap with Retroactive Advantage Correction (RAC): each pending slow completion is queued, aged through a non-negative kernel, and reinjected as a clipped residual into the next optimiser step's advantage. We prove that under an unbiased clipped importance ratio, the cumulative RAC correction is exactly unbiased when the effective delay kernel reinjects all of its mass, and carries a bias linear in the unreinjected fraction otherwise; at the no-delay identity kernel it reduces to V-trace. On a tabular Markov decision process (MDP) proof-of-concept, RAC reduces the closed-form policy bias by up to 47.9x at the two-slow-channel configuration, beating wait-for-slow at lower wall-clock cost. RAC integrates with PPO and GRPO through a two-line reward-manager patch.

Problem

• Reinforcement Learning from Human Feedback
• asynchronous rewards
• delayed feedback
• reward delay
• synchronous-reward assumption

Innovation

• Retroactive Advantage Correction
• delay-aware RLHF
• V-trace bias correction
• asynchronous reward
• closed-form policy bias