MIRA: A Novel Framework for Fusing Modalities in Medical RAG

2025-07-10

AI Summary

Multimodal large language models (MLLMs) suffer from factual inaccuracies in medical question answering and report generation, undermining clinical reliability. Existing retrieval-augmented generation (RAG) approaches face dual challenges: insufficient retrieval leads to missing critical information, while excessive retrieval introduces noise and disrupts the model’s intrinsic reasoning by overwhelming it with external knowledge. To address these issues, we propose a dynamic RAG framework featuring a “Re-think and Re-rank” module that adaptively determines the optimal number of retrieved contexts based on query semantics and image embeddings. Furthermore, we integrate query rewriting with multimodal embedding alignment to enable synergistic verification between the model’s internal knowledge and external medical resources. Evaluated on multiple public medical visual question answering and report generation benchmarks, our method significantly improves factual accuracy and generation quality, achieving state-of-the-art performance. The implementation is publicly available.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Multimodal Large Language Models (MLLMs) have significantly advanced AI-assisted medical diagnosis, but they often generate factually inconsistent responses that deviate from established medical knowledge. Retrieval-Augmented Generation (RAG) enhances factual accuracy by integrating external sources, but it presents two key challenges. First, insufficient retrieval can miss critical information, whereas excessive retrieval can introduce irrelevant or misleading content, disrupting model output. Second, even when the model initially provides correct answers, over-reliance on retrieved data can lead to factual errors. To address these issues, we introduce the Multimodal Intelligent Retrieval and Augmentation (MIRA) framework, designed to optimize factual accuracy in MLLM. MIRA consists of two key components: (1) a calibrated Rethinking and Rearrangement module that dynamically adjusts the number of retrieved contexts to manage factual risk, and (2) A medical RAG framework integrating image embeddings and a medical knowledge base with a query-rewrite module for efficient multimodal reasoning. This enables the model to effectively integrate both its inherent knowledge and external references. Our evaluation of publicly available medical VQA and report generation benchmarks demonstrates that MIRA substantially enhances factual accuracy and overall performance, achieving new state-of-the-art results. Code is released at https://github.com/mbzuai-oryx/MIRA.

Problem

• Addresses factual inconsistencies in medical MLLM responses
• Optimizes retrieval balance to avoid irrelevant content
• Reduces over-reliance on retrieved data causing errors

Innovation

• Dynamic retrieval adjustment for factual risk
• Image embeddings with medical knowledge base
• Query-rewrite module for multimodal reasoning