GazeXPErT: An Expert Eye-tracking Dataset for Interpretable and Explainable AI in Oncologic FDG-PET/CT Scans

2026-02-25

AI Summary

This study addresses the shortage of clinical experts and the limited interpretability and workflow integration of existing AI models in oncological FDG-PET/CT image analysis. To bridge this gap, we present the first 4D dataset integrating high-frequency (60 Hz) eye-tracking data with 346 FDG-PET/CT scans, synchronously capturing radiologists’ visual search trajectories and lesion annotations, structured in COCO format and comprising 9,030 gaze-to-lesion correspondence sequences. This resource enables research in visual grounding, intention prediction, and interpretable modeling. Experimental results demonstrate that incorporating expert gaze information improves nnUNet’s segmentation DICE score from 0.6008 to 0.6819. Furthermore, a Vision Transformer model aligns 74.95% of predicted fixations closer to tumors, achieving an intention prediction accuracy of 67.53% (AUROC: 0.747).

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Web Mining and Content Analysis: Large pretrained models with web data
• Security and Privacy: Data transparency and provenance

Abstract

[18F]FDG-PET/CT is a cornerstone imaging modality for tumor staging and treatment response assessment across many cancer types, yet expert reader shortages necessitate more efficient diagnostic aids. While standalone AI models for automatic lesion segmentation exist, clinical translation remains hindered by concerns about interpretability, explainability, reliability, and workflow integration. We present GazeXPErT, a 4D eye-tracking dataset capturing expert search patterns during tumor detection and measurement on 346 FDG-PET/CT scans. Each study was read by a trainee and a board-certified nuclear medicine or radiology specialist using an eye-tracking-enabled annotation platform that simulates routine clinical reads. From 3,948 minutes of raw 60Hz eye-tracking data, 9,030 unique gaze-to-lesion trajectories were extracted, synchronized with PET/CT image slices, and rendered in COCO-style format for multiple machine learning applications. Baseline validation experiments demonstrate that a 3D nnUNet tumor segmentation model achieved superior performance when incorporating expert gaze patterns versus without (DICE score 0.6819 versus 0.6008), and that vision transformers trained on sequential gaze and PET/CT images can improve dynamic lesion localization (74.95% predicted gaze point closer to tumor) and expert intention prediction (Accuracy 67.53% and AUROC 0.747). GazeXPErT is a valuable resource designed to explore multiple machine learning problems beyond these baseline experiments, which include and are not limited to, visual grounding or causal reasoning, clinically explainable feature augmentation, human-computer interaction, human intention prediction or understanding, and expert gaze-rewarded modeling approaches to AI in oncologic FDG-PET/CT imaging.

Problem

• interpretability
• explainability
• FDG-PET/CT
• expert eye-tracking
• clinical AI

Innovation

• eye-tracking
• explainable AI
• FDG-PET/CT
• gaze-guided learning
• clinical interpretability