IMCBench: A benchmark for multimodal LLMs in Image-grounded Medical Conversations

2026-06-26

AI Summary

Existing medical AI evaluation benchmarks either lack image-supported multi-turn dialogue capabilities or are confined to single-round question answering, making it difficult to comprehensively assess model performance in realistic clinical settings. This work proposes the first image-guided, multi-turn physician-patient dialogue benchmark, integrating real clinical images with synthetically generated electronic health records (EHRs) to construct a multidimensional evaluation framework encompassing safety, factual accuracy, and uncertainty expression. The study further introduces an expert-calibrated LLM-as-Jury mechanism for fine-grained automated scoring. Experimental results show that Claude Opus 4.6 achieves the best overall performance (3.61/5), yet all models exhibit markedly reduced safety in scenarios involving malignancies or rare diseases. Ablation studies confirm that both visual inputs and EHR context are critical for safe clinical reasoning.

Abstract

Recent advances in large language models and vision-language models have enabled reasoning over multimodal data, offering opportunities for clinical applications such as decision support and triaging. However, existing medical AI benchmarks are fragmented: some support multi-turn dialogues but lack images, while others provide multimodal inputs but focus on single-turn QA tasks. To address this gap, we introduce IMCBench, an image-grounded, multi-turn medical conversation benchmark that pairs real, publicly available clinical images with synthetic patient profiles to simulate realistic patient-clinician interactions. Each conversation is evaluated across three clinical dimensions: safety, accuracy, and appropriate use of uncertainty in diagnosis. We benchmark eight multimodal frontier models across four model families (Claude, GPT, Nova, and Llama), scoring each on a 1-5 scale using LLM-as-Jury scoring calibrated against expert clinician annotations. Our results show that Claude Opus 4.6 achieves the highest overall score (3.61), followed by Claude Sonnet 4.6 (3.30) and GPT-5.2 (3.29), though no model dominates all dimensions and safety degrades for both malignant and rare conditions ($Δ$ = -0.27 each). Ablation studies further reveal that both visual input and EHR context contribute to safe guidance (safety drops of 0.18 and 0.23 on average when each is removed), with stronger models leveraging visual features more effectively. Together, these findings demonstrate that accurate clinical description does not guarantee safe patient guidance, motivating the need for multi-dimensional evaluation frameworks in medical AI.

Problem

• multimodal LLMs
• medical conversation
• image-grounded
• benchmark
• clinical safety

Innovation

• multimodal LLMs
• medical conversation benchmark
• image-grounded dialogue
• clinical safety evaluation
• LLM-as-Jury