Trust but Verify: Mitigating Medical Hallucinations via Post-Hoc Adversarial Auditing and Multi-Agent Feedback Loops

2026-06-12

AI Summary

This study addresses the critical issue of hallucinations in large language models (LLMs) within clinical settings, where erroneous outputs—such as recommending contraindicated medications—pose serious patient safety risks. To mitigate this without modifying the underlying model, the authors propose a model-agnostic, multi-agent “trust but verify” framework that integrates adversarial auditing with real-time regulatory data. Through coordinated interaction among five specialized agents, the system detects and corrects hazardous responses at inference time. Evaluated on adversarial datasets, the approach reduces hallucination error rates by an average of 53% and shifts recommendation scores from −0.25 (unsafe) to near 0.0 (appropriately abstaining), thereby enhancing clinical safety. The work also introduces novel evaluation metrics—including label accuracy, point-wise scoring, and component fidelity—to comprehensively assess safety and regulatory compliance.

Abstract

Large Language Models (LLMs) are increasingly deployed in healthcare settings, yet their tendency to hallucinate poses risks when clinical decisions are involved. This study examine whether LLMs recommend recently banned or withdrawn pharmaceuticals when answering clinical questions and tests an agent-based method for reducing such errors. We developed a five-agent "Trust but Verify" system using a single LLM backbone. To measure regulatory knowledge obsolescence, we created an adversarial dataset of 103 clinical MCQs where historically correct answers now refer to banned substances. This scale ensures statistical significance across various therapeutic classes. We evaluated three open-access model families (GPT-OSS, Llama-3, Falcon-3) under vanilla and agentic conditions. Performance was measured via pointwise score, label accuracy, Hallucination Error Rate (HER), and Component Fidelity (CF) score. We also observed clinical safety regression in proprietary models. In default configurations, all models showed high hallucination rates, consistently selecting banned drugs that matched training data patterns. Our proposed agentic architecture reduced HER by approximately 53% across models. Pointwise scores shifted from -0.25 (unsafe recommendation) toward 0.0 (appropriate refusal). The safety audit intercepted dangerous outputs even when models' parametric knowledge favored the banned substance. The proposed multi-agent framework offers a model-agnostic method for enforcing regulatory compliance that prioritizes patient safety over fluent text generation. Our work demonstrates a practical approach for deploying autonomous AI systems in safety-critical healthcare settings. It shows how real-time regulatory data can be integrated into LLM pipelines to support clinical decision-making.

Problem

• medical hallucinations
• regulatory compliance
• clinical safety
• banned pharmaceuticals
• LLM reliability

Innovation

• adversarial auditing
• multi-agent feedback loops
• medical hallucination mitigation
• regulatory compliance
• model-agnostic safety