Visual Reasoning Evaluation of Grok, Deepseek Janus, Gemini, Qwen, Mistral, and ChatGPT

2025-02-23

AI Summary

Existing LLM multimodal evaluation frameworks are limited to single-image reasoning, neglecting critical dimensions such as cross-image contextual understanding, reasoning stability, and uncertainty calibration. Method: We introduce the first comprehensive benchmark for multi-image visual reasoning, featuring three novel components: (1) joint multi-image reasoning tasks with explicit cross-image context modeling; (2) entropy-driven metrics for quantifying reasoning consistency; and (3) position reordering perturbations to jointly assess rejection capability and positional bias sensitivity. Contribution/Results: Experiments reveal substantial performance disparities: ChatGPT-o1 achieves 82.5% overall accuracy (70.0% rejection accuracy), QVQ-72B-Preview attains 85.5% rejection accuracy, while Janus exhibits high entropy (0.787–0.839) and pronounced positional bias. Our benchmark establishes a new paradigm for trustworthy, fine-grained evaluation of multimodal LLMs—advancing beyond isolated image analysis toward robust, context-aware, and uncertainty-aware visual reasoning assessment.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Knowledge Representation and Reasoning: Qualitative Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Traditional evaluations of multimodal large language models (LLMs) have been limited by their focus on single-image reasoning, failing to assess crucial aspects like contextual understanding, reasoning stability, and uncertainty calibration. This study addresses these limitations by introducing a novel benchmark that integrates multi-image reasoning tasks with rejection-based evaluation and positional bias detection. To evaluate these dimensions, we further introduce entropy as a novel metric for quantifying reasoning consistency across reordered answer variants. We applied this benchmark to assess Grok 3, ChatGPT-4o, ChatGPT-o1, Gemini 2.0 Flash Experimental, DeepSeek Janus models, Qwen2.5-VL-72B-Instruct, QVQ-72B-Preview, and Pixtral 12B across eight visual reasoning tasks, including difference spotting and diagram interpretation. Our findings reveal ChatGPT-o1 leading in overall accuracy (82.5%) and rejection accuracy (70.0%), closely followed by Gemini 2.0 Flash Experimental (70.8%). QVQ-72B-Preview demonstrated superior rejection accuracy (85.5%). Notably, Pixtral 12B (51.7%) showed promise in specific domains, while Janus models exhibited challenges in bias and uncertainty calibration, reflected in low rejection accuracies and high entropy scores. High entropy scores in Janus models (Janus 7B: 0.8392, Janus 1B: 0.787) underscore their susceptibility to positional bias and unstable reasoning, contrasting with the low entropy and robust reasoning of ChatGPT models. The study further demonstrates that model size is not the sole determinant of performance, as evidenced by Grok 3 underperformance despite its substantial parameter count. By employing multi-image contexts, rejection mechanisms, and entropy-based consistency metrics, this benchmark sets a new standard for evaluating multimodal LLMs, enabling a more robust and reliable assessment of next-generation AI systems.

Problem

• Evaluates multimodal LLMs' contextual understanding
• Introduces multi-image reasoning benchmark
• Assesses reasoning stability and uncertainty calibration

Innovation

• Multi-image reasoning benchmark
• Rejection-based evaluation method
• Entropy for reasoning consistency