GazeVaLM: A Multi-Observer Eye-Tracking Benchmark for Evaluating Clinical Realism in AI-Generated X-Rays

2026-04-13

AI Summary

This study addresses the lack of a multi-observer eye-tracking benchmark for evaluating the clinical realism of AI-generated chest X-rays, which has hindered quantitative assessment of agreement between human experts and artificial intelligence in image perception and authenticity judgment. To bridge this gap, the authors construct a multimodal benchmark dataset comprising 960 eye-tracking records—including fixations, scanpaths, and saliency maps—from 16 radiologists diagnosing and discriminating the authenticity of 60 real and diffusion-model-generated chest X-rays, along with structured diagnostic labels. Concurrently, predictions from six state-of-the-art multimodal large language models on the same tasks are collected. This benchmark uniquely integrates human eye-movement behavior, clinical decisions, and AI outputs, enabling systematic comparison of human–AI alignment in diagnostic accuracy, authenticity detection, and uncertainty quantification, thereby extending the visual Turing test to clinical imaging.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Explainable AI (XAI) for Human Understanding

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Security and Privacy: Large-scale security measurements

Abstract

We introduce GazeVaLM, a public eye-tracking dataset for studying clinical perception during chest radiograph authenticity assessment. The dataset comprises 960 gaze recordings from 16 expert radiologists interpreting 30 real and 30 synthetic chest X-rays (generated by diffusion based generative AI) under two conditions: diagnostic assessment and real-fake classification (Visual Turing test). For each image-observer pair, we provide raw gaze samples, fixation maps, scanpaths, saliency density maps, structured diagnostic labels, and authenticity judgments. We extend the protocol to 6 state-of-the-art multimodal LLMs, releasing their predicted diagnoses, authenticity labels, and confidence scores under matched conditions - enabling direct human-AI comparison at both decision and uncertainty levels. We further provide analyses of gaze agreement, inter-observer consistency, and benchmarking of radiologists versus LLMs in diagnostic accuracy and authenticity detection. GazeVaLM supports research in gaze modeling, clinical decision-making, human-AI comparison, generative image realism assessment, and uncertainty quantification. By jointly releasing visual attention data, clinical labels, and model predictions, we aim to facilitate reproducible research on how experts and AI systems perceive, interpret, and evaluate medical images. The dataset is available at https://huggingface.co/datasets/davidcwong/GazeVaLM.

Problem

• clinical realism
• AI-generated X-rays
• eye-tracking
• Visual Turing test
• medical image authenticity

Innovation

• eye-tracking
• multimodal LLMs
• generative AI
• clinical realism
• Visual Turing test