ViPO: Visual Preference Optimization at Scale

2026-04-27

AI Summary

Existing visual preference optimization methods struggle to learn consistent preferences due to noisy, low-resolution, and imbalanced preference data with limited prompt diversity. To address these limitations, this work proposes Poly-DPO, an algorithm that enhances robustness to noise by dynamically modulating model confidence through polynomial terms. Additionally, we introduce ViPO, a large-scale, high-quality visual preference dataset comprising one million image pairs at 1024px resolution and 300,000 video pairs at 720p+, spanning diverse categories and prompts. Experiments show that Poly-DPO improves GenEval scores by 6.87 (SD1.5) and 2.32 (SDXL) over Diffusion-DPO on Pick-a-Pic V2. Notably, when trained on the high-quality ViPO data, standard DPO achieves superior performance, underscoring the critical synergy between algorithmic design and data quality in preference optimization.

Abstract

While preference optimization is crucial for improving visual generative models, how to effectively scale this paradigm remains largely unexplored. Current open-source preference datasets contain conflicting preference patterns, where winners excel in some dimensions but underperform in others. Naively optimizing on such noisy datasets fails to learn preferences, hindering effective scaling. To enhance robustness against noise, we propose Poly-DPO, which extends the DPO objective with an additional polynomial term that dynamically adjusts model confidence based on dataset characteristics, enabling effective learning across diverse data distributions. Beyond biased patterns, existing datasets suffer from low resolution, limited prompt diversity, and imbalanced distributions. To facilitate large-scale visual preference optimization by tackling data bottlenecks, we construct ViPO, a massive-scale preference dataset with 1M image pairs at 1024px across five categories and 300K video pairs at 720p+ across three categories. State-of-the-art generative models and diverse prompts ensure reliable preference signals with balanced distributions. Remarkably, when applying Poly-DPO to our high-quality dataset, the optimal configuration converges to standard DPO. This convergence validates dataset quality and Poly-DPO's adaptive nature: sophisticated optimization becomes unnecessary with sufficient data quality, yet remains valuable for imperfect datasets. We validate our approach across visual generation models. On noisy datasets like Pick-a-Pic V2, Poly-DPO achieves 6.87 and 2.32 gains over Diffusion-DPO on GenEval for SD1.5 and SDXL, respectively. For ViPO, models achieve performance far exceeding those trained on existing open-source preference datasets. These results confirm that addressing both algorithmic adaptability and data quality is essential for scaling visual preference optimization.

Problem

• visual preference optimization
• preference dataset
• scaling
• noise robustness
• data quality

Innovation

• Preference Optimization
• Poly-DPO
• ViPO Dataset
• Visual Generation
• Scalable Learning