Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Models

2025-02-20

AI Summary

Current vision-language models (VLMs) lack a unified, authoritative benchmark for evaluating multimodal reward modeling, hindering rigorous assessment of their alignment with human preferences. To address this, we introduce Multimodal RewardBench—the first expert-annotated benchmark specifically designed for VLM reward modeling. It spans six core dimensions: correctness, preference, knowledge, reasoning, safety, and visual question answering, comprising 5,211 high-quality triplets (prompt-response-reference). Our systematic evaluation framework integrates multi-model response sampling, multi-dimensional prompt engineering, and fine-grained human annotation. Experiments reveal that state-of-the-art models—including Gemini 1.5 Pro and Claude 3.5 Sonnet—achieve only 72% overall accuracy on this benchmark, with pronounced deficiencies in reasoning and safety, underscoring the critical need for standardized evaluation. The benchmark is publicly released to foster standardization and healthy advancement in multimodal reward modeling.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Reward models play an essential role in training vision-language models (VLMs) by assessing output quality to enable aligning with human preferences. Despite their importance, the research community lacks comprehensive open benchmarks for evaluating multimodal reward models in VLMs. To address this gap, we introduce Multimodal RewardBench, an expert-annotated benchmark covering six domains: general correctness, preference, knowledge, reasoning, safety, and visual question-answering. Our dataset comprises 5,211 annotated (prompt, chosen response, rejected response) triplets collected from various VLMs. In evaluating a range of VLM judges, we find that even the top-performing models, Gemini 1.5 Pro and Claude 3.5 Sonnet, achieve only 72% overall accuracy. Notably, most models struggle in the reasoning and safety domains. These findings suggest that Multimodal RewardBench offers a challenging testbed for advancing reward model development across multiple domains. We release the benchmark at https://github.com/facebookresearch/multimodal_rewardbench.

Problem

• Lack of comprehensive benchmarks for multimodal reward models
• Need for holistic evaluation across multiple domains
• Challenges in reasoning and safety domains for VLMs

Innovation

• Multimodal RewardBench benchmark
• Expert-annotated dataset
• Comprehensive evaluation domains