AtomiMed: Hierarchical Atomic Fact-Checking for Universal Clinical-Aware Medical Report Evaluation

2026-06-30

AI Summary

This work addresses the limitations of existing evaluation metrics for medical report generation, which rely on surface-level n-gram overlap and fail to capture clinical factual accuracy, often overlooking critical diagnostic errors. To this end, the authors propose AtomiMed, a novel framework that introduces hierarchical atomic clinical fact modeling by decomposing reports into disease entities and their attribute descriptions. It further incorporates a multi-agent proxy cross-validation mechanism that simulates radiologist peer review, enabling decoupled assessment of diagnostic detection and descriptive accuracy. Accompanied by an open-source toolkit, MRGEvalKit, and a multimodal benchmark, OmniMRG-Bench, the framework establishes the first cross-modal, clinically aware evaluation paradigm. Experimental results demonstrate that AtomiMed significantly outperforms current metrics in multiple expert reader studies, achieving substantially higher correlation with radiologists’ judgments.

Abstract

Traditional metrics for Medical Report Generation (MRG) predominantly rely on surface-level n-gram overlap, which fails to capture clinical factual accuracy and often overlooks catastrophic diagnostic errors. We address this fundamental limitation by proposing \textbf{AtomiMed}, a universal, modality-agnostic evaluation framework that decomposes complex medical narratives into a standardized, multi-level hierarchy of Atomic Clinical Facts, encompassing Disease-level entities and Attribute-level descriptors, including location, morphology, and severity. By implementing an Agentic Cross-Verification loop between ground-truth and predicted reports, AtomiMed simulates a multi-radiologist peer-review process to verify clinical consistency, thus enabling the decoupled assessment of diagnostic detection and descriptive accuracy. To facilitate standardized evaluation, we introduce \textbf{MRGEvalKit}, an open-source toolkit for automated hierarchical extraction, and curate \textbf{OmniMRG-Bench}, a comprehensive multi-modal benchmark covering X-ray, CT, MRI, and Ultrasound. Extensive experiments on multiple expert-annotated reader studies demonstrate that AtomiMed achieves significantly higher correlation with human radiologist judgment compared to traditional and model-based metrics. Our code are release at https://github.com/Venn2336/MRGEvalkit

Problem

• Medical Report Generation
• factual accuracy
• clinical evaluation
• diagnostic errors
• evaluation metrics

Innovation

• Atomic Clinical Facts
• Hierarchical Evaluation
• Agentic Cross-Verification
• Medical Report Generation
• Clinical Factual Accuracy