AEGPO: Adaptive Entropy-Guided Policy Optimization for Diffusion Models

2026-02-06

AI Summary

This work addresses a key limitation in existing reinforcement learning from human feedback (RLHF) methods—such as GRPO—for diffusion model optimization: their reliance on static, uniform sampling that overlooks the varying learning utility across prompts and denoising timesteps, as well as the dynamic nature of critical exploration opportunities. To overcome this, the paper introduces attention entropy as a dual proxy signal. Globally, changes in entropy (ΔEntropy) dynamically allocate rollout budgets; locally, entropy peaks (Entropy(t)) identify high-uncertainty timesteps to guide targeted exploration. This two-level adaptive policy optimization mechanism significantly accelerates convergence in text-to-image generation and achieves superior alignment performance compared to standard GRPO variants.

Technology Category

• Search and Optimization: Learning to Search
• Computer Vision: Diffusion Models for Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Reinforcement learning from human feedback (RLHF) shows promise for aligning diffusion and flow models, yet policy optimization methods such as GRPO suffer from inefficient and static sampling strategies. These methods treat all prompts and denoising steps uniformly, ignoring substantial variations in sample learning value as well as the dynamic nature of critical exploration moments. To address this issue, we conduct a detailed analysis of the internal attention dynamics during GRPO training and uncover a key insight: attention entropy can serve as a powerful dual-signal proxy. First, across different samples, the relative change in attention entropy ({\Delta}Entropy), which reflects the divergence between the current policy and the base policy, acts as a robust indicator of sample learning value. Second, during the denoising process, the peaks of absolute attention entropy (Entropy(t)), which quantify attention dispersion, effectively identify critical timesteps where high-value exploration occurs. Building on this observation, we propose Adaptive Entropy-Guided Policy Optimization (AEGPO), a novel dual-signal, dual-level adaptive optimization strategy. At the global level, AEGPO uses {\Delta}Entropy to dynamically allocate rollout budgets, prioritizing prompts with higher learning value. At the local level, it exploits the peaks of Entropy(t) to guide exploration selectively at critical high-dispersion timesteps rather than uniformly across all denoising steps. By focusing computation on the most informative samples and the most critical moments, AEGPO enables more efficient and effective policy optimization. Experiments on text-to-image generation tasks demonstrate that AEGPO significantly accelerates convergence and achieves superior alignment performance compared to standard GRPO variants.

Problem

• Reinforcement Learning from Human Feedback
• Policy Optimization
• Diffusion Models
• Sampling Strategy
• Attention Entropy

Innovation

• attention entropy
• adaptive sampling
• policy optimization
• diffusion models
• reinforcement learning from human feedback