Human-Feedback Efficient Reinforcement Learning for Online Diffusion Model Finetuning

2024-10-07

arXiv.org

AI Summary

Existing online fine-tuning methods for Stable Diffusion suffer from inefficient human feedback utilization, reliance on predefined reward functions, or large-scale offline-pretrained reward models. Method: This paper proposes HERO—a human-feedback-driven optimization framework that requires neither predefined reward functions nor offline reward models, instead leveraging real-time human feedback directly. HERO introduces two novel mechanisms: (i) feedback-aligned representation learning, which models human intent in the latent space, and (ii) feedback-guided image generation, enabling efficient convergence. Contribution/Results: HERO achieves multi-task generalization with only 500 online feedback instances. In body anomaly correction, it improves feedback efficiency by 4×. It also significantly outperforms baseline methods on reasoning, counting, personalized generation, and NSFW content suppression tasks.

Technology Category

• Search and Optimization: Learning to Search
• Humans and AI: Learning Human Values and Preferences
• Computer Vision: Diffusion Models for Vision

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Controllable generation through Stable Diffusion (SD) fine-tuning aims to improve fidelity, safety, and alignment with human guidance. Existing reinforcement learning from human feedback methods usually rely on predefined heuristic reward functions or pretrained reward models built on large-scale datasets, limiting their applicability to scenarios where collecting such data is costly or difficult. To effectively and efficiently utilize human feedback, we develop a framework, HERO, which leverages online human feedback collected on the fly during model learning. Specifically, HERO features two key mechanisms: (1) Feedback-Aligned Representation Learning, an online training method that captures human feedback and provides informative learning signals for fine-tuning, and (2) Feedback-Guided Image Generation, which involves generating images from SD's refined initialization samples, enabling faster convergence towards the evaluator's intent. We demonstrate that HERO is 4x more efficient in online feedback for body part anomaly correction compared to the best existing method. Additionally, experiments show that HERO can effectively handle tasks like reasoning, counting, personalization, and reducing NSFW content with only 0.5K online feedback.

Problem

• Improves fidelity, safety, and alignment in Stable Diffusion fine-tuning.
• Reduces reliance on costly predefined reward models or datasets.
• Enhances efficiency in online human feedback utilization.

Innovation

• Online human feedback for model fine-tuning
• Feedback-Aligned Representation Learning mechanism
• Feedback-Guided Image Generation for faster convergence