Noisy-Space Policy Gradient for Diffusion Policies in Offline Reinforcement Learning

2026-09-07

AI Summary

本文通过引入带噪声的动作价值函数和KL正则化策略改进方法，解决了扩散策略在离线强化学习中的集成难题。

Abstract

Diffusion policies offer a powerful and expressive parameterization for continuous control. Yet, their integration with reinforcement learning remains conceptually and algorithmically challenging. In this work, we address this gap by introducing a noisy-space action-value (Q-)function that assigns values to diffusion latents through the distribution of executed actions induced by the denoising process. We show that this construction admits a precise semantic interpretation and derive a noisy-space policy gradient (NSPG) that optimizes noisy latents using only clean action-space value estimates. Building on this result, we formulate a KL-regularized policy improvement over noisy latents and show that the resulting objective admits a diffusion-compatible regression form, avoiding backpropagation through the denoising process. Empirical results on state-based D4RL benchmarks and vision-based OGBench tasks demonstrate that the proposed noisy-space objective provides a principled and effective basis for training diffusion policies in offline reinforcement learning. Project webpage: https://mahmoud-selim.github.io/NSPG/

Problem

• diffusion policies
• offline reinforcement learning
• noisy-space policy gradient

Innovation

• Noisy-Space Q-function
• Noisy-Space Policy Gradient (NSPG)
• KL-regularized policy improvement
• Diffusion policies