CheXpercept: A Benchmark for Evaluating Expert-Level Lesion Perception in Chest X-rays

2026-06-18

AI Summary

Current vision-language models exhibit limited fine-grained lesion perception in chest X-ray analysis, hindering reliable clinical assessment. To address this gap, this work proposes the first structured evaluation framework explicitly designed for expert-level lesion awareness, emulating radiologists’ cognitive workflow—from coarse detection and refined contour delineation to semantic attribute extraction. The authors construct a multi-tiered benchmark comprising 2,100 chest radiographs and 10,400 question-answer pairs via semi-automatic methods, rigorously validated by six medical experts to ensure clinical fidelity. A systematic evaluation of 14 state-of-the-art models reveals that while existing approaches perform moderately in coarse localization, they critically lack deeper perceptual capabilities; notably, even models specifically trained on medical data show no significant advantage over general-purpose counterparts.

Abstract

The evaluation of vision-language models (VLMs) for chest X-ray (CXR) analysis has largely been limited to disease-presence classification without visual grounding. Such evaluations fail to verify the expert-level lesion perception necessary to ensure the clinical reliability of VLMs. To address these limitations, we introduce CheXpercept, a sequential, multi-level perception benchmark that mirrors a radiologist's cognitive workflow across coarse-level detection, fine-level contour evaluation and revision, and semantic-level attribute extraction. To ensure high clinical fidelity at scale, we construct the dataset using a semi-automated generation pipeline paired with a review by six medical experts. CheXpercept contains 10,400 QA items derived from 2,100 CXRs, covering seven clinically critical pulmonary and cardiac lesions. To demonstrate the current landscape of VLM perception, we benchmark 14 general and medical VLMs on CheXpercept. The models achieve adequate performance only at the coarse level, with accuracy degrading precipitously on deeper visual tasks. Notably, medical VLMs show almost no perceptual advantage over their general-domain counterparts, highlighting a systemic flaw in current domain adaptation. The code and dataset will be publicly available.

Problem

• vision-language models
• chest X-ray
• lesion perception
• clinical reliability
• visual grounding

Innovation

• vision-language models
• lesion perception
• chest X-ray benchmark
• multi-level visual reasoning
• clinical evaluation