Diffusion Guidance Is a Controllable Policy Improvement Operator

2025-05-29

AI Summary

This work addresses the complexity and instability inherent in offline reinforcement learning (RL) methods that rely on explicit value functions. We propose a novel value-free paradigm for policy optimization, wherein classifier-free guidance (CFG) from diffusion models is reformulated as a tunable policy improvement operator—a theoretical connection established for the first time between diffusion-based guidance and policy improvement. Based on this insight, we introduce CFGRL, a framework trained via target-conditioned behavioral cloning in a supervised manner. Policy improvement is achieved solely by adjusting the guidance scale, eliminating the need for value estimation or iterative optimization. Empirical results across diverse offline RL benchmarks demonstrate that increasing the guidance weight consistently and stably improves performance; remarkably, without additional training, CFGRL significantly outperforms baseline methods, yielding a “free” performance gain.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

At the core of reinforcement learning is the idea of learning beyond the performance in the data. However, scaling such systems has proven notoriously tricky. In contrast, techniques from generative modeling have proven remarkably scalable and are simple to train. In this work, we combine these strengths, by deriving a direct relation between policy improvement and guidance of diffusion models. The resulting framework, CFGRL, is trained with the simplicity of supervised learning, yet can further improve on the policies in the data. On offline RL tasks, we observe a reliable trend -- increased guidance weighting leads to increased performance. Of particular importance, CFGRL can operate without explicitly learning a value function, allowing us to generalize simple supervised methods (e.g., goal-conditioned behavioral cloning) to further prioritize optimality, gaining performance for"free"across the board.

Problem

• Combines policy improvement with diffusion model guidance
• Enhances performance without explicit value function learning
• Generalizes supervised methods to prioritize optimality

Innovation

• Combines policy improvement with diffusion guidance
• Trains simply with supervised learning methods
• Operates without explicit value function learning