HPSv3++: Scaling Reward Models Across the Full Spectrum of Diffusion Model Capabilities

2026-06-12

AI Summary

Existing reward models struggle to adapt to the shifting quality discrimination demands of diffusion models across varying capability levels and reinforcement learning (RL) training stages, resulting in limited generalization. To address this, this work proposes the HPSv3++ framework, which introduces a novel conditional signal mechanism that jointly accounts for generative capability and RL stage, alongside a two-dimensional preference dataset, HPDv3++. The method employs a two-stage training pipeline incorporating standard deviation–driven unsupervised guidance and data-aware orthogonal gradient projection to maintain continuous alignment of the reward model within dynamically evolving systems. Experiments demonstrate that HPSv3++ outperforms HPSv3 by 9.8% and 5.5% on HPDv3 and GenAI-Bench, respectively, achieves accuracies of 79.1% and 88.1% on the newly curated HPDv3++, and significantly improves GenEval scores across multiple text-to-image models.

Abstract

Reward models guide text-to-image (T2I) systems toward outputs aligned with human preferences. However, typical reward models such as HPSv3 are trained on pre-annotated data from earlier T2I models, without accounting for quality discriminative shifts arising from evolving model capabilities and reinforcement learning (RL) iterations, limiting their broader applicability. In this work, we propose HPSv3++, a reward model framework that elevates the HPSv3 model for varying T2I model capabilities and their RL iteration changes across the full capability-iteration spectrum. Specifically, we first introduce HPDv3++, a 212K dual-dimension preference dataset annotated for text fidelity and aesthetic quality using a recent high-capability (Qwen-Image) model with human supervision. We then propose a two-stage training framework. Stage 1 employs data-aware orthogonal gradient projection to incorporate diverse aesthetic perception from HPDv3++ while preserving the original effective human preference knowledge in HPSv3. Stage 2 further leverages unlabeled data from T2I models spanning different capability levels and RL iterations, and introduces a joint capability-iterations conditioned signal for the reward model together with a standard deviation-driven unsupervised guidance mechanism, strengthening reward model across the capability-iteration spectrum. HPSv3++ achieves state-of-the-art preference prediction, outperforming HPSv3 9.8% on HPDv3, 5.5% on GenAI-Bench, while achieving 79.1%/88.1% on our proposed HPDv3++. When used for T2I RL training, it consistently improves GenEval scores across diverse T2I models, demonstrating its wide-range capabilities. The code is available at https://github.com/PlantPotatoOnMoon/HPSv3-PlusPlus.

Problem

• reward models
• text-to-image generation
• human preference alignment
• model capability evolution
• reinforcement learning iterations

Innovation

• reward modeling
• text-to-image generation
• preference dataset
• orthogonal gradient projection
• capability-iteration spectrum