FBOS-RL: Feedback-Driven Bi-Objective Synergistic Reinforcement Learning

2026-05-18

AI Summary

This work addresses the challenge that traditional reinforcement learning methods, such as GRPO, struggle to generate high-quality rollouts when tasks exceed the model’s current capabilities, resulting in weak gradient signals and training stagnation. To overcome this limitation, the authors propose a feedback-driven dual-objective cooperative reinforcement learning framework that, for the first time, leverages environmental feedback to guide exploration while jointly optimizing two complementary objectives: exploitation-oriented policy alignment (EPA) and exploration-oriented capability cultivation (ECC). This mechanism dynamically balances exploration and exploitation, effectively breaking through training bottlenecks. Experimental results demonstrate that, under the same number of rollouts, the proposed method achieves significantly faster training convergence and superior final performance compared to GRPO and feedback-based baselines, while maintaining higher policy entropy and lower gradient norms.

Abstract

Reinforcement learning has become a cornerstone for aligning and unlocking the reasoning capabilities of large-scale models. At its core, the training loop of GRPO and its variants alternates between rollout sampling and policy update. Unlike supervised learning, where each gradient step is anchored to an explicit ground-truth target, the optimal gradient direction for updating model parameters in this setting is not known a priori; the high-quality rollouts drawn during the sampling stage therefore act as the implicit "teacher" that guides every parameter update. However, GRPO adopt a simple sampling scheme that conditions all rollouts on the same original prompt. When a task lies beyond the policy model's current capability, this sampling scheme rarely yields a high-quality rollout, leaving the policy model without a meaningful gradient direction when updating its parameters, which causes training to stall. To address this issue, we propose FBOS-RL, a Feedback-Driven Bi-Objective Synergistic reinforcement learning framework. Specifically, we let the model perform Feedback-Guided Exploration Enhancement based on the feedback provided by the environment, and on top of this we design two mutually reinforcing training objectives: Exploitation-oriented Policy Alignment(EPA) and Exploration-oriented Capability Cultivation(ECC). Extensive experiments demonstrate that EPA and ECC can mutually reinforce each other, forming a positive flywheel effect that significantly improves both the training efficiency and the final performance ceiling of reinforcement learning. Specifically, under an identical number of rollouts, FBOS-RL learns substantially faster than GRPO and feedback-based baselines and ultimately attains a higher performance ceiling, while exhibiting higher policy entropy and lower gradient norms throughout training.

Problem

• reinforcement learning
• rollout sampling
• policy update
• training stagnation
• gradient direction

Innovation

• Feedback-Driven Exploration
• Bi-Objective Synergy
• Policy Alignment
• Capability Cultivation
• Reinforcement Learning