ReportQA: QA-Based Radiology Report Evaluation

2026-06-12

AI Summary

This work proposes ReportQA, a novel framework that introduces a question-answering paradigm to radiology report evaluation, addressing the limitations of existing metrics—namely their weak clinical relevance, inability to capture fine-grained medical entities and attributes, and heavy reliance on manual annotations. Guided by radiologists, the approach constructs a clinical knowledge tree and leverages large language models to extract structured information from reports. High-quality question-answer pairs are then generated through a template-driven, two-stage filtering process, with evaluation based on QA accuracy (QAScore). Experiments demonstrate that QAScore exhibits strong agreement with radiologists’ judgments at a fine-grained level and significantly outperforms current methods. The study further reveals inherent limitations of contemporary vision–language models in clinical representation learning and validates the efficacy of question-driven reasoning for report assessment.

Abstract

Radiology report evaluation is essential for advancing automated report generation. Natural language generation metrics have limited clinical relevance. Clinical efficacy (CE) metrics evaluate important medical findings, but focus mainly on presence and cover only a limited set of entities. Due to heavy reliance on manual annotations, it is difficult for CE metrics to extend clinical entities or attributes. In clinical practice, radiology reports serve as a medium for information transfer. Clinicians use them to perform downstream diagnostic tasks without directly inspecting images. Based on this insight, we propose ReportQA, a clinical-related and flexible radiology report evaluation framework, supporting detailed quantitative analysis of radiology report generation systems. We first collect datasets covering multiple imaging modalities and anatomical regions. We then construct knowledge trees of clinical entities and attributes with radiologist guidance, and use large language models (LLMs) to extract structured information from raw reports. Next, we generate QA pairs from predefined templates and apply quality control through self-filtering and report-based filtering. During evaluation, the report is treated as context, and an LLM acts as a judge model to answer the QA pairs. Based on the resulting QA accuracy, we introduce QAScore metric. Compared with existing metrics, QAScore shows better alignment with radiologist judgments. Experiments on multiple state-of-the-art vision-language models reveal that current report-based inference paradigms struggle to learn fine-grained clinical representations and exhibit strong negative prior biases. In contrast, question-driven inference provides a more effective alternative. For reproducibility and extensibility, we release the knowledge trees, structured reports, and QA pairs, along with the pipeline code for QA construction and evaluation.

Problem

• radiology report evaluation
• clinical relevance
• natural language generation metrics
• clinical efficacy metrics
• automated report generation

Innovation

• ReportQA
• radiology report evaluation
• question-answering
• large language models
• clinical knowledge tree