MApLe: Multi-instance Alignment of Diagnostic Reports and Large Medical Images

2026-04-15

AI Summary

Current vision-language models struggle to accurately align critical textual descriptions in medical reports with subtle yet clinically significant local regions in images. This work proposes a multi-task, multi-instance fine-grained image-text alignment approach that decouples anatomical structures from pathological findings. By introducing an anatomy-conditioned image patch encoder and integrating specially trained text embeddings with a multi-instance alignment mechanism, the method achieves precise correspondence between sentence-level diagnostic descriptions and relevant image regions. Evaluated across multiple downstream tasks, the proposed approach significantly outperforms state-of-the-art baselines, effectively enabling accurate mapping between multiple lesions described in free-text radiology reports and their corresponding visual regions.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

In diagnostic reports, experts encode complex imaging data into clinically actionable information. They describe subtle pathological findings that are meaningful in their anatomical context. Reports follow relatively consistent structures, expressing diagnostic information with few words that are often associated with tiny but consequential image observations. Standard vision language models struggle to identify the associations between these informative text components and small locations in the images. Here, we propose "MApLe", a multi-task, multi-instance vision language alignment approach that overcomes these limitations. It disentangles the concepts of anatomical region and diagnostic finding, and links local image information to sentences in a patch-wise approach. Our method consists of a text embedding trained to capture anatomical and diagnostic concepts in sentences, a patch-wise image encoder conditioned on anatomical structures, and a multi-instance alignment of these representations. We demonstrate that MApLe can successfully align different image regions and multiple diagnostic findings in free-text reports. We show that our model improves the alignment performance compared to state-of-the-art baseline models when evaluated on several downstream tasks. The code is available at https://github.com/cirmuw/MApLe.

Problem

• vision-language alignment
• medical imaging
• diagnostic reports
• multi-instance learning
• fine-grained localization

Innovation

• vision-language alignment
• multi-instance learning
• medical image-text alignment
• anatomical-diagnostic disentanglement
• patch-wise encoding