Medical thinking with multiple images

2026-04-14

AI Summary

This work addresses the limitations of existing medical visual question answering (VQA) models, which predominantly rely on single-image inputs and thus fail to support the multi-view image reasoning required in clinical practice. To bridge this gap, the authors introduce MedThinkVQA—the first high-density multi-image medical VQA benchmark, featuring an average of 6.62 images per case—augmented with expert annotations, intermediate supervision, and step-level evaluation protocols. Through systematic evaluation of leading large models and error attribution analysis, they find that even state-of-the-art closed-source and open-source models achieve only 57.2% and 52.2% accuracy, respectively. Over 70% of errors stem from misinterpretation of individual images and failures in integrating evidence across views, underscoring the critical role of reliable visual grounding in multi-image diagnostic reasoning.

Technology Category

• Computer Vision: Multi-modal Vision
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning
• Machine Learning: Multi-instance/Multi-view Learning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Large language models perform well on many medical QA benchmarks, but real clinical reasoning often requires integrating evidence across multiple images rather than interpreting a single view. We introduce MedThinkVQA, an expert-annotated benchmark for thinking with multiple images, where models must interpret each image, combine cross-view evidence, and answer diagnostic questions with intermediate supervision and step-level evaluation. The dataset contains 8,067 cases, including 720 test cases, with an average of 6.62 images per case, substantially denser than prior work, whose expert-level benchmarks use at most 1.43 images per case. On the test set, the best closed-source models, Claude-4.6-Opus, Gemini-3-Pro, and GPT-5.2-xhigh, reach only 57.2%, 55.3%, and 54.9% accuracy, while GPT-5-mini and GPT-5-nano reach 39.7% and 30.8%. Strong open-source models lag behind, led by Qwen3.5-397B-A17B at 52.2% and Qwen3.5-27B at 50.6%. Further analysis identifies grounded multi-image reasoning as the main bottleneck: models often fail to extract, align, and compose evidence across views before higher-level inference can help. Providing expert single-image cues and cross-image summaries improves performance, whereas replacing them with self-generated intermediates reduces accuracy. Step-level analysis shows that over 70% of errors arise from image reading and cross-view integration. Scaling results further show that additional inference-time computation helps only when visual grounding is already reliable; when early evidence extraction is weak, longer reasoning yields limited or unstable gains and can amplify misread cues. These results suggest that the key challenge is not reasoning length alone, but reliable mechanisms for grounding, aligning, and composing distributed evidence across real-world multimodal clinical inputs.

Problem

• multi-image reasoning
• medical visual question answering
• clinical evidence integration
• visual grounding
• multimodal reasoning

Innovation

• multi-image reasoning
• visual grounding
• clinical VQA
• step-level evaluation
• evidence composition