Sample-Efficient Diffusion-based Reinforcement Learning with Critic Guidance

2026-05-28

AI Summary

Existing diffusion-based reinforcement learning methods struggle to balance exploration and exploitation: sampling-based policy optimization exhibits strong exploration but insufficient utilization of Q-values, while gradient-based approaches often converge to single modes and lack diversity. To address this challenge, this work proposes Critic-Guided Policy Optimization (CGPO), which introduces, for the first time, a training-free critic-guidance mechanism during the denoising process. This mechanism steers action generation toward high-value regions and serves as a regression target, effectively unifying efficient exploration with effective exploitation while preserving diversity. CGPO achieves state-of-the-art performance across five MuJoCo benchmark tasks and demonstrates, for the first time, successful deployment on a real-world Franka robotic arm grasping task, validating the practical efficacy of diffusion policies in real-world reinforcement learning settings.

Abstract

Recent advances in reinforcement learning (RL) have achieved great successes by leveraging the multimodality and exploration capability of diffusion policies. Among these approaches, one representative branch focuses on the sampling-based policy optimization. This design enables better exploration capability of the diffusion model, particularly at the beginning of training, but suffer from low exploitation in Q-value information, resulting in a slow policy convergence. Another branch pays attention to gradient-based policy optimization, which sufficiently exploits the gradient of the Q function yet tends to collapse into a unimodal policy with low diversity. To address this issue, we propose CGPO, \textbf{C}ritic-\textbf{G}uided diffusion \textbf{P}olicy \textbf{O}ptimization, which effectively balances exploration and exploitation with the training-free guidance technique integrated into the denoising process of diffusion policy. Concretely, CGPO steers action generation toward high-value regions defined by the critic network and uses the guided actions as regression objectives. In this manner, CGPO reduces the time required to obtain high-quality actions and improves final performance with better balance between the exploration-exploitation tradeoff. We validate the effectiveness of CGPO on 5 MuJoCo locomotion tasks, and CGPO achieves state-of-the-art performance compared with existing diffusion-based RL methods. Notably, CGPO is the first success to incorporate diffusion policy into real-world RL, with its superior performance on Franka robot arm grasping tasks. Our official page is released at https://dingsht.tech/cgpo-webpage.

Problem

• diffusion-based reinforcement learning
• exploration-exploitation tradeoff
• policy optimization
• multimodality
• sample efficiency

Innovation

• diffusion policy
• critic guidance
• sample-efficient reinforcement learning
• exploration-exploitation tradeoff
• real-world robotics