Visual Reasoning Evaluation of Grok, Deepseek Janus, Gemini, Qwen, Mistral, and ChatGPT
Existing LLM multimodal evaluation frameworks are limited to single-image reasoning, neglecting critical dimensions such as cross-image contextual understanding, reasoning stability, and uncertainty calibration. Method: We introduce the first comprehensive benchmark for multi-image visual reasoning, featuring three novel components: (1) joint multi-image reasoning tasks with explicit cross-image context modeling; (2) entropy-driven metrics for quantifying reasoning consistency; and (3) position reordering perturbations to jointly assess rejection capability and positional bias sensitivity. Contribution/Results: Experiments reveal substantial performance disparities: ChatGPT-o1 achieves 82.5% overall accuracy (70.0% rejection accuracy), QVQ-72B-Preview attains 85.5% rejection accuracy, while Janus exhibits high entropy (0.787–0.839) and pronounced positional bias. Our benchmark establishes a new paradigm for trustworthy, fine-grained evaluation of multimodal LLMs—advancing beyond isolated image analysis toward robust, context-aware, and uncertainty-aware visual reasoning assessment.