DiffExp: Efficient Exploration in Reward Fine-tuning for Text-to-Image Diffusion Models

2025-02-19

AI Summary

To address inefficient exploration and slow convergence caused by online sampling in reward-based fine-tuning of text-to-image diffusion models, this paper proposes a dual-path exploration mechanism. First, it dynamically modulates the classifier-free guidance scale to enhance generative diversity; second, it introduces random reweighting sampling at the text-phrase level to precisely concentrate sampling on high-reward regions. This is the first work to jointly model dynamic guidance scheduling and phrase-level weight perturbation, significantly improving sample efficiency. Evaluated on mainstream frameworks—including DDPO and AlignProp—the method achieves an average 18.7% improvement in reward score under identical sample budgets, reduces training steps by 32%, and demonstrates faster convergence and superior generalization across multiple benchmarks.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Sampling/Simulation-based Search
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Fine-tuning text-to-image diffusion models to maximize rewards has proven effective for enhancing model performance. However, reward fine-tuning methods often suffer from slow convergence due to online sample generation. Therefore, obtaining diverse samples with strong reward signals is crucial for improving sample efficiency and overall performance. In this work, we introduce DiffExp, a simple yet effective exploration strategy for reward fine-tuning of text-to-image models. Our approach employs two key strategies: (a) dynamically adjusting the scale of classifier-free guidance to enhance sample diversity, and (b) randomly weighting phrases of the text prompt to exploit high-quality reward signals. We demonstrate that these strategies significantly enhance exploration during online sample generation, improving the sample efficiency of recent reward fine-tuning methods, such as DDPO and AlignProp.

Problem

• Enhance sample diversity in reward fine-tuning
• Improve convergence speed of diffusion models
• Optimize text-to-image model performance efficiently

Innovation

• Dynamic classifier-free guidance scaling
• Random text prompt phrase weighting
• Enhanced exploration in online generation