MR. Judge: Multimodal Reasoner as a Judge

2025-05-19

AI Summary

Weak reasoning capabilities and poor interpretability of multimodal large language models (MLLMs) when employed as evaluative judges hinder their reliability in preference learning. To address this, we propose MR. Judge—a novel paradigm that reconfigures MLLMs as discriminative adjudicators capable of multi-step, multi-dimensional, multiple-choice-style reasoning, replacing conventional direct scoring. Our method introduces a reasoning-driven multimodal discrimination framework, integrating reverse response synthesis and textual reasoning distillation to construct high-quality discriminative data without human annotation. On VL-RewardBench, MR. Judge-7B outperforms GPT-4o by 9.9%; in MM-Vet, it achieves up to 7.7% improvement under inference-time scaling. These results demonstrate substantial gains in discrimination accuracy and interpretability for both RLHF and inference-time scaling scenarios.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Preferences
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

The paradigm of using Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) as evaluative judges has emerged as an effective approach in RLHF and inference-time scaling. In this work, we propose Multimodal Reasoner as a Judge (MR. Judge), a paradigm for empowering general-purpose MLLMs judges with strong reasoning capabilities. Instead of directly assigning scores for each response, we formulate the judgement process as a reasoning-inspired multiple-choice problem. Specifically, the judge model first conducts deliberate reasoning covering different aspects of the responses and eventually selects the best response from them. This reasoning process not only improves the interpretibility of the judgement, but also greatly enhances the performance of MLLM judges. To cope with the lack of questions with scored responses, we propose the following strategy to achieve automatic annotation: 1) Reverse Response Candidates Synthesis: starting from a supervised fine-tuning (SFT) dataset, we treat the original response as the best candidate and prompt the MLLM to generate plausible but flawed negative candidates. 2) Text-based reasoning extraction: we carefully design a data synthesis pipeline for distilling the reasoning capability from a text-based reasoning model, which is adopted to enable the MLLM judges to regain complex reasoning ability via warm up supervised fine-tuning. Experiments demonstrate that our MR. Judge is effective across a wide range of tasks. Specifically, our MR. Judge-7B surpasses GPT-4o by 9.9% on VL-RewardBench, and improves performance on MM-Vet during inference-time scaling by up to 7.7%.

Problem

• Enhancing MLLM judges with reasoning for better evaluations
• Formulating judgement as reasoning-based multiple-choice problem
• Automating annotation via synthetic flawed response generation

Innovation

• MLLMs as judges with reasoning capabilities
• Reasoning-inspired multiple-choice judgement process
• Automatic annotation via reverse response synthesis