Seeing Through Experts Eyes A Foundational Vision Language Model Trained on Radiologists Gaze and Reasoning

2026-04-15

AI Summary

This study addresses the clinical misalignment of current medical imaging AI systems, which often overlook critical findings due to their failure to model radiologists’ visual scanning patterns and diagnostic reasoning. To bridge this gap, the authors propose a novel approach that leverages expert eye-tracking data as a supervisory signal during vision-language pretraining, introducing a gaze-guided spatiotemporal attention mechanism that enforces adherence to systematic diagnostic protocols. Trained on a large-scale multimodal dataset comprising over 30,000 eye-tracking keyframes, 230,000 medical images, and 780,000 question-answer pairs, the model significantly improves performance in radiology report generation, lesion localization, and visual question answering—achieving higher accuracy and greater alignment with expert interpretations. Moreover, it produces traceable and interpretable diagnostic evidence chains, thereby facilitating safe and efficient human-AI collaboration in clinical workflows.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Explainable AI (XAI) for Human Understanding

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Large scale vision language models have shown promise in automating chest Xray interpretation, yet their clinical utility remains limited by a gap between model outputs and radiologist reasoning. Most systems optimize for semantic information without emulating how experts visually examine medical images, often overlooking critical findings or diverging from established diagnostic workflows. Radiologists follow structured protocols (e.g., the ABCDEF approach) that ensure all clinically relevant regions are systematically examined, reducing missed findings and supporting reliable diagnostic reasoning. We introduce GazeX, a vision language model that leverages radiologists' eye tracking data as a behavioral prior to model expert diagnostic reasoning. By incorporating gaze trajectories and fixation patterns into pretraining, GazeX learns to follow the spatial and temporal structure of radiologist attention and integrates observations in a clinically meaningful sequence. Using a curated dataset of over 30,000 gaze key frames from five radiologists, we demonstrate that GazeX produces more accurate, interpretable, and expert consistent outputs across radiology report generation, disease grounding, and visual question answering, utilizing 231,835 radiographic studies, 780,014 question answer pairs, and 1,162 image sentence pairs with bounding boxes. Unlike autonomous reporting systems, GazeX produces verifiable evidence artifacts, including inspection trajectories and finding linked localized regions, enabling efficient human verification and safe human AI collaboration. Learning through expert eyes provides a practical route toward more trustworthy, explainable, and diagnostically robust AI systems for radiology and beyond.

Problem

• vision language model
• radiologist reasoning
• clinical utility
• gaze tracking
• diagnostic workflow

Innovation

• gaze tracking
• vision-language model
• radiologist reasoning
• diagnostic workflow
• explainable AI