Auto-Rubric as Reward: From Implicit Preferences to Explicit Multimodal Generative Criteria

2026-05-08

AI Summary

Existing multimodal generative models often struggle to align with human preferences due to scalar or pairwise rewards that fail to capture the multidimensional nature of human judgment, leading to reward hacking and evaluation bias. This work proposes shifting reward modeling from implicit weight optimization to an explicit, rubric-based decomposition approach, externalizing human preferences into verifiable and composable scoring criteria across distinct quality dimensions. By integrating vision-language models, prompt engineering, and a novel Rubric Policy Optimization (RPO) algorithm, the method enables both zero-shot deployment and few-shot fine-tuning. Evaluated on text-to-image generation and image editing tasks, it significantly outperforms conventional reward models and VLM-based evaluators, achieving notable improvements in alignment reliability and data efficiency.

Abstract

Aligning multimodal generative models with human preferences demands reward signals that respect the compositional, multi-dimensional structure of human judgment. Prevailing RLHF approaches reduce this structure to scalar or pairwise labels, collapsing nuanced preferences into opaque parametric proxies and exposing vulnerabilities to reward hacking. While recent Rubrics-as-Reward (RaR) methods attempt to recover this structure through explicit criteria, generating rubrics that are simultaneously reliable, scalable, and data-efficient remains an open problem. We introduce Auto-Rubric as Reward (ARR), a framework that reframes reward modeling from implicit weight optimization to explicit, criteria-based decomposition. Before any pairwise comparison, ARR externalizes a VLM's internalized preference knowledge as prompt-specific rubrics, translating holistic intent into independently verifiable quality dimensions. This conversion of implicit preference structure into inspectable, interpretable constraints substantially suppresses evaluation biases including positional bias, enabling both zero-shot deployment and few-shot conditioning on minimal supervision. To extend these gains into generative training, we propose Rubric Policy Optimization (RPO), which distills ARR's structured multi-dimensional evaluation into a robust binary reward, replacing opaque scalar regression with rubric-conditioned preference decisions that stabilize policy gradients. On text-to-image generation and image editing benchmarks, ARR-RPO outperforms pairwise reward models and VLM judges, demonstrating that explicitly externalizing implicit preference knowledge into structured rubrics achieves more reliable, data-efficient multimodal alignment, revealing that the bottleneck is the absence of a factorized interface, not a deficit of knowledge.

Problem

• multimodal generative models
• human preferences
• reward modeling
• preference alignment
• evaluation bias

Innovation

• Auto-Rubric as Reward
• Rubric Policy Optimization
• multimodal alignment
• explicit criteria
• reward modeling