BRACE: Anchored Bellman-Residual Correction for Stale Critics in Asynchronous RL

2026-09-09

AI Summary

为解决异步强化学习中价值函数滞后问题，提出BRACE方法，通过锚定贝尔曼残差修正策略，提高模型性能和训练效率。

Abstract

Asynchronous reinforcement learning has become the standard way to scale training for language models, but the resulting policy lag biases the critic toward the stale behavior policy. Existing work on asynchronous LLM training corrects the actor and leaves this bias unaddressed, while the off-policy value correction of classical RL does not carry over to long-horizon agentic tasks, since a short correction horizon leaves the regression target free of the reward and a long one lets the product of importance ratios drift exponentially with the trajectory length. We propose BRACE, an anchored Bellman-residual correction for stale value models. BRACE bounds the correction horizon to a prefix of policy tokens and anchors a constant-weight Monte-Carlo tail beyond it, which separates policy correction from reward propagation. BRACE improves mean@1 on BrowseComp-Plus by $2.4\%$ over the strongest baseline, runs $2.46\times$ faster per step than synchronous training, and remains stable $50$ updates off-policy.

Problem

• asynchronous reinforcement learning
• policy lag
• critic bias
• off-policy value correction
• importance ratios

Innovation

• Anchored Bellman-Residual Correction
• Stale Critics
• Asynchronous Reinforcement Learning
• Policy Lag