Automatic Fine-grained Segmentation-assisted Report Generation

2025-07-22

AI Summary

This study addresses the low reliability and poor interpretability of end-to-end clinical imaging report generation. We propose ASaRG, the first framework integrating fine-grained anatomical segmentation maps—generated by a domain-specific radiology model—with the LLaVA multimodal architecture. By performing intermediate-layer feature concatenation, ASaRG achieves precise alignment between generated report text and underlying anatomical structures, thereby enhancing traceability and clinical credibility of model reasoning. On the CE benchmark, ASaRG achieves a 2.77% higher F1 score than the LLaVA baseline (p < 0.001), outperforming COMG and ORID by 6.98% and 6.28%, respectively. The framework alleviates radiologists’ workload while delivering verifiable, clinically grounded second opinions.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Argumentation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Reliable end-to-end clinical report generation has been a longstanding goal of medical ML research. The end goal for this process is to alleviate radiologists' workloads and provide second opinions to clinicians or patients. Thus, a necessary prerequisite for report generation models is a strong general performance and some type of innate grounding capability, to convince clinicians or patients of the veracity of the generated reports. In this paper, we present ASaRG ( extbf{A}utomatic extbf{S}egmentation- extbf{a}ssisted extbf{R}eport extbf{G}eneration), an extension of the popular LLaVA architecture that aims to tackle both of these problems. ASaRG proposes to fuse intermediate features and fine-grained segmentation maps created by specialist radiological models into LLaVA's multi-modal projection layer via simple concatenation. With a small number of added parameters, our approach achieves a +0.89% performance gain ($p=0.012$) in CE F1 score compared to the LLaVA baseline when using only intermediate features, and +2.77% performance gain ($p<0.001$) when adding a combination of intermediate features and fine-grained segmentation maps. Compared with COMG and ORID, two other report generation methods that utilize segmentations, the performance gain amounts to 6.98% and 6.28% in F1 score, respectively. ASaRG is not mutually exclusive with other changes made to the LLaVA architecture, potentially allowing our method to be combined with other advances in the field. Finally, the use of an arbitrary number of segmentations as part of the input demonstrably allows tracing elements of the report to the corresponding segmentation maps and verifying the groundedness of assessments. Our code will be made publicly available at a later date.

Problem

• Generate reliable clinical reports to reduce radiologists' workload
• Improve report accuracy by integrating fine-grained segmentation maps
• Ensure model grounding for verifiable assessments in generated reports

Innovation

• Fuses segmentation maps into LLaVA's projection layer
• Uses specialist radiological models for fine-grained segmentation
• Enables traceability of report elements to segmentation maps