MMPCBench: Benchmarking Multimodal Large Language Models on Proactive Critique of Flawed Inputs

2026-08-29

AI Summary

研究提出MMPCBench框架，通过定义主动批评能力来评估多模态大语言模型在识别、分析和修正用户输入错误方面的能力，揭示了模型在处理视觉异常时的弱点。

Abstract

As Multimodal Large Language Models (MLLMs) evolve into sophisticated interactive assistants, their reliability depends not only on following instructions but also on validating them. We define Proactive Critique as the model's autonomous ability to identify, analyze and fix faulty user inputs without extra prompts. However, evaluations mainly test models under ideal circumstances or simple refusal behaviors, largely ignoring active error processing. To fill this gap, we propose MMPCBench, a comprehensive framework for evaluating MLLMs' proactive critique competence. It features a fine-grained taxonomy of 4 primary error types spanning 12 subcategories, ranging from cross-modal contradictions to missing visual premises. We adopt a hierarchical evaluation protocol to measure models' error detection, diagnosis and resolution performance, and apply alignment-aware metrics to assess the coherence between internal reasoning and final responses. Tests on 14 mainstream MLLMs show obvious weaknesses in proactive critique, especially in dealing with subtle visual anomalies. Notably, we identify a pervasive "consistency gap": reasoning models can often correctly identify and analyze errors during internal reasoning yet suppress these valid insights in final outputs to prioritize response compliance. The code and data is available at https://github.com/ALIENS32/MMPCBench.

Problem

• Multimodal Large Language Models
• Proactive Critique
• Error Processing

Innovation

• Proactive Critique
• Multimodal Large Language Models
• Error Detection and Resolution
• Alignment-Aware Metrics
• Consistency Gap