Visual Preference Optimization with Rubric Rewards

2026-04-14

AI Summary

This work addresses the limitations of existing visual preference optimization methods, which rely on coarse-grained feedback or off-policy perturbations and struggle to support fine-grained visual reasoning. The authors propose rDPO, a novel framework that introduces, for the first time, an instance-level scoring rubric mechanism. This mechanism defines core and auxiliary criteria for each image-instruction pair, enabling the construction of an offline rubric pool subsequently used for online policy data generation and response scoring. By integrating rubric-guided prompt engineering and rubric-based response filtering, rDPO significantly enhances the quality and task relevance of preference data. Experimental results demonstrate that rDPO achieves performance close to GPT-5.4 on established reward modeling benchmarks, with a downstream macro-average score of 82.69 and a composite scalability score of 61.01, both surpassing current state-of-the-art baselines.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Knowledge Representation and Reasoning: Preferences
• Humans and AI: Learning Human Values and Preferences

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

The effectiveness of Direct Preference Optimization (DPO) depends on preference data that reflect the quality differences that matter in multimodal tasks. Existing pipelines often rely on off-policy perturbations or coarse outcome-based signals, which are not well suited to fine-grained visual reasoning. We propose rDPO, a preference optimization framework based on instance-specific rubrics. For each image-instruction pair, we create a checklist-style rubric of essential and additional criteria to score responses from any possible policies. The instruction-rubric pool is built offline and reused during the construction of on-policy data. On public reward modeling benchmarks, rubric-based prompting massively improves a 30B-A3B judge and brings it close to GPT-5.4. On public downstream benchmarks, rubric-based filtering raises the macro average to 82.69, whereas outcome-based filtering drops it to 75.82 from 81.14. When evaluating scalability on a comprehensive benchmark, rDPO achieves 61.01, markedly outperforming the style-constrained baseline (52.36) and surpassing the 59.48 base model. Together, these results show that visual preference optimization benefits from combining on-policy data construction with instance-specific criterion-level feedback.

Problem

• Visual Preference Optimization
• Multimodal Tasks
• Fine-grained Visual Reasoning
• Preference Data
• Rubric-based Evaluation

Innovation

• rubric-based preference optimization
• visual reasoning
• on-policy data construction
• multimodal alignment
• fine-grained feedback