Towards Responsible Multimodal Medical Reasoning via Context-Aligned Vision-Language Models

2026-04-09

AI Summary

This work addresses the susceptibility of existing medical vision-language models to hallucinated diagnoses in radiology due to overreliance on a single modality. To mitigate this, the authors propose a context-aligned reasoning framework that enforces consensus among multiple clinical evidence sources—including radiomics statistics, explainable activation maps, and lexical semantic cues—before generating a diagnostic statement. The framework guides a frozen vision-language model to produce structured outputs explicitly incorporating supporting evidence, uncertainty estimates, limitations, and safety disclaimers. Rather than merely fusing modalities, the approach employs a context-validation mechanism to enable synergistic multi-evidence reasoning, substantially enhancing reliability and transparency. Experiments on chest X-ray data demonstrate improved performance (AUC increased from 0.918 to 0.925), a 78% reduction in hallucination-related keywords (from 1.14 to 0.25 per report), more concise outputs (19.4 to 15.3 words), and well-calibrated uncertainty with stable confidence scores.

Technology Category

• Computer Vision: Multi-modal Vision
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Medical vision-language models (VLMs) show strong performance on radiology tasks but often produce fluent yet weakly grounded conclusions due to over-reliance on a dominant modality. We introduce a context-aligned reasoning framework that enforces agreement across heterogeneous clinical evidence before generating diagnostic conclusions. The proposed approach augments a frozen VLM with structured contextual signals derived from radiomic statistics, explainability activations, and vocabulary-grounded semantic cues. Instead of producing free-form responses, the model generates structured outputs containing supporting evidence, uncertainty estimates, limitations, and safety notes. We observe that auxiliary signals alone provide limited benefit; performance gains emerge only when these signals are integrated through contextual verification. Experiments on chest X-ray datasets demonstrate that context alignment improves discriminative performance (AUC 0.918 to 0.925) while maintaining calibrated uncertainty. The framework also substantially reduces hallucinated keywords (1.14 to 0.25) and produces more concise reasoning explanations (19.4 to 15.3 words) without increasing model confidence (0.70 to 0.68). Cross-dataset evaluation on CheXpert further reveals that modality informativeness significantly influences reasoning behavior. These results suggest that enforcing multi-evidence agreement improves both reliability and trustworthiness in medical multimodal reasoning, while preserving the underlying model architecture.

Problem

• medical vision-language models
• multimodal reasoning
• hallucination
• modality imbalance
• diagnostic reliability

Innovation

• context-aligned reasoning
• vision-language models
• multimodal medical reasoning
• structured output generation
• evidence agreement