CheXthought: A global multimodal dataset of clinical chain-of-thought reasoning and visual attention for chest X-ray interpretation

2026-04-29

AI Summary

Current vision–language models struggle to accurately capture radiologists’ clinical reasoning and visual attention mechanisms, resulting in chest X-ray interpretations that lack both accuracy and interpretability. This work introduces a large-scale multimodal dataset comprising over 100,000 clinical chains-of-thought and more than 6.6 million synchronized visual attention annotations from 501 radiologists across 71 countries, covering over 50,000 chest radiographs. For the first time, this resource systematically records and structures expert cognitive processes and visual search strategies. Models trained on this dataset significantly outperform existing approaches in pathology classification, visual faithfulness, temporal reasoning, and hallucination suppression. Moreover, they can predict diagnostic discrepancies between human–human and human–AI pairs, thereby enhancing diagnostic transparency and reliability.

Abstract

Chest X-ray interpretation is one of the most frequently performed diagnostic tasks in medicine and a primary target for AI development, yet current vision--language models are primarily trained on datasets of paired images and reports, not the cognitive processes and visual attention that underlie clinical reasoning. Here, we present CheXthought, a global, multimodal resource containing 103,592 chain-of-thought reasoning traces and 6,609,082 synchronized visual attention annotations across 50,312 multi-read chest X-rays from 501 radiologists in 71 countries. Our analysis reveals clinical reasoning patterns in how experts deploy distinct visual search strategies, integrate clinical context, and communicate uncertainty. We demonstrate the clinical utility of CheXthought across four dimensions. First, CheXthought reasoning significantly outperforms state--of--the--art vision--language model chain-of-thought in factual accuracy and spatial grounding. Second, visual attention data used as an inference--time hint recovers missed findings and significantly reduces hallucinations. Third, models trained on CheXthought data achieve significantly stronger pathology classification, visual faithfulness, temporal reasoning and uncertainty communication. Fourth, leveraging CheXthought's multi-reader annotations, we predict both human--human and human--AI disagreement directly from an image, enabling transparent communication of case difficulty, uncertainty and model reliability. These findings establish CheXthought as a resource for advancing multimodal clinical reasoning and the development of more transparent, interpretable vision--language models.

Problem

• chest X-ray interpretation
• clinical reasoning
• visual attention
• vision-language models
• multimodal dataset

Innovation

• chain-of-thought reasoning
• visual attention
• multimodal clinical reasoning
• vision-language models
• uncertainty communication