HPSv3: Towards Wide-Spectrum Human Preference Score

2025-08-05

AI Summary

Existing text-to-image evaluation metrics suffer from misalignment with human perception due to limitations in data coverage, feature representation, and loss design. To address this, we propose a human-centered evaluation paradigm. First, we construct HPDv3—the first million-scale human preference dataset spanning the full quality spectrum. Second, we design a fine-grained preference ranking method grounded in vision-language models (VLMs), incorporating an uncertainty-aware pairwise ranking loss. Third, we introduce Chain-of-Human-Preference (CoHP), an iterative optimization framework that improves generation quality without requiring additional human annotations. Extensive experiments demonstrate that our evaluation system, HPSv3, achieves high robustness across diverse generative models and real-world scenarios, while exhibiting strong correlation with human judgments. The code and dataset are publicly released to advance human-aligned evaluation standards.

Technology Category

• Humans and AI: Learning Human Values and Preferences
• Computer Vision: Large Vision Models
• Machine Learning: Learning Preferences or Rankings

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Evaluating text-to-image generation models requires alignment with human perception, yet existing human-centric metrics are constrained by limited data coverage, suboptimal feature extraction, and inefficient loss functions. To address these challenges, we introduce Human Preference Score v3 (HPSv3). (1) We release HPDv3, the first wide-spectrum human preference dataset integrating 1.08M text-image pairs and 1.17M annotated pairwise comparisons from state-of-the-art generative models and low to high-quality real-world images. (2) We introduce a VLM-based preference model trained using an uncertainty-aware ranking loss for fine-grained ranking. Besides, we propose Chain-of-Human-Preference (CoHP), an iterative image refinement method that enhances quality without extra data, using HPSv3 to select the best image at each step. Extensive experiments demonstrate that HPSv3 serves as a robust metric for wide-spectrum image evaluation, and CoHP offers an efficient and human-aligned approach to improve image generation quality. The code and dataset are available at the HPSv3 Homepage.

Problem

• Evaluating text-to-image alignment with human perception
• Overcoming limited data coverage in human-centric metrics
• Improving image generation quality without extra data

Innovation

• Wide-spectrum dataset with 1.08M text-image pairs
• VLM-based model with uncertainty-aware ranking loss
• Chain-of-Human-Preference for iterative image refinement