D-SPEAR: Dual-Stream Prioritized Experience Adaptive Replay for Stable Reinforcement Learninging Robotic Manipulation

2026-03-28

AI Summary

This work addresses the instability and performance degradation commonly observed in reinforcement learning for contact-rich, long-horizon robotic manipulation tasks, which often stem from conventional experience replay strategies failing to account for the distinct data requirements of policy and value networks. To mitigate this, the authors propose D-SPEAR, a decoupled dual-stream experience replay framework: the value network leverages prioritized experience replay for efficient learning, while the policy network samples transitions with low temporal difference (TD) error to ensure stable optimization. An adaptive anchoring mechanism dynamically adjusts the sampling strategy based on the coefficient of variation of TD errors, and Huber loss is incorporated to enhance robustness to heterogeneous reward scales. Evaluated on robosuite tasks such as Block-Lifting and Door-Opening, D-SPEAR consistently outperforms baseline algorithms—including SAC, TD3, and DDPG—in both final performance and training stability.

Technology Category

• Intelligent Robots: Learning & Optimization for ROB
• Multiagent Systems: Adversarial Agents
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Robotic manipulation remains challenging for reinforcement learning due to contact-rich dynamics, long horizons, and training instability. Although off-policy actor-critic algorithms such as SAC and TD3 perform well in simulation, they often suffer from policy oscillations and performance collapse in realistic settings, partly due to experience replay strategies that ignore the differing data requirements of the actor and the critic. We propose D-SPEAR: Dual-Stream Prioritized Experience Adaptive Replay, a replay framework that decouples actor and critic sampling while maintaining a shared replay buffer. The critic leverages prioritized replay for efficient value learning, whereas the actor is updated using low-error transitions to stabilize policy optimization. An adaptive anchor mechanism balances uniform and prioritized sampling based on the coefficient of variation of TD errors, and a Huber-based critic objective further improves robustness under heterogeneous reward scales. We evaluate D-SPEAR on challenging robotic manipulation tasks from the robosuite benchmark, including Block-Lifting and Door-Opening. Results demonstrate that D-SPEAR consistently outperforms strong off-policy baselines, including SAC, TD3, and DDPG, in both final performance and training stability, with ablation studies confirming the complementary roles of the actorside and critic-side replay streams.

Problem

• robotic manipulation
• reinforcement learning
• training instability
• experience replay
• off-policy learning

Innovation

• Dual-Stream Replay
• Prioritized Experience Replay
• Actor-Critic Decoupling
• Adaptive Sampling
• Robust Reinforcement Learning