Synergistic Perception-Reasoning Governance: Grounding Medical MLLMs with Verifiable Anatomical Evidence

2026-06-30

AI Summary

Medical multimodal large language models are prone to generating hallucinations during inference that contradict imaging evidence, undermining clinical credibility. This work proposes a training-free evidence-injection framework that employs dual-path intervention: it modulates visual activations using regions of interest extracted by MedSAM and anchors text generation by mapping anatomical coordinates into verifiable semantic tokens. Task-aware dynamic routing enables modality-specific collaborative regulation. The approach achieves, for the first time, training-agnostic joint control over perception and reasoning by directly embedding anatomical evidence—eliminating the need for additional training or post-processing. Evaluated across five medical datasets, the method significantly outperforms baselines, improving closed-ended question accuracy by up to 6% and reducing open-ended hallucinations by 35%.

Abstract

Multimodal large language models (MLLMs) show strong promise for clinical VQA and radiology report generation, yet inference-time hallucinations still undermine trustworthy use: models can produce fluent conclusions that conflict with imaging evidence. Existing mitigation strategies typically rely on additional training, external retrieval/knowledge bases, or multi-stage post-hoc verification, which increases cost and pipeline complexity and often generalizes poorly across models and tasks.To address this, we propose a holistic, training-free evidence-injection framework that systematically mitigates hallucinations through dual-side evidence injection. By leveraging ROI priors acquired using MedSAM in our implementation, we recalibrate the visual perception trajectory via ROI-guided activation modulation while anchoring the textual reasoning trajectory by mapping anatomical coordinates into discrete semantic tokens as verifiable external memory. Then we introduce a task-aware dynamic router to select modality-specific interventions based on task semantics, balancing perceptual grounding and linguistic fluency. We conduct systematic evaluations on 2 tasks and 5 datasets using \texttt{LLaVA-1.5-7B}, \texttt{LLaVA-Med-1.5-7B}, \texttt{Qwen3-VL-8B/32B}, and \texttt{InternVL-3.5-8B/38B}. Controlled ablations and visualizations further validate the framework, which consistently outperforms baselines across medical benchmarks, improving close-ended accuracy by up to $\sim\mathbf{6}\%\uparrow$ and reducing open-ended hallucinations by $\sim\mathbf{35}\%\downarrow$. The code has been made available on GitHub: \href{https://github.com/Henry991115/SPRG}{\textcolor{blue}{https://github.com/Henry991115/SPRG}}.

Problem

• hallucination
• multimodal large language models
• medical VQA
• radiology report generation
• trustworthy AI

Innovation

• hallucination mitigation
• evidence injection
• multimodal LLMs
• anatomical grounding
• training-free framework