Assessing and Mitigating Medical Knowledge Drift and Conflicts in Large Language Models

2025-05-12

AI Summary

This work addresses knowledge drift and internal inconsistencies in large language models (LLMs) arising from dynamically evolving clinical guidelines. We introduce DriftMedQA—the first medical temporal reliability evaluation framework—comprising 4,290 clinically grounded scenarios, which systematically exposes critical deficiencies in mainstream LLMs, including failure to reject outdated recommendations and susceptibility to endorsing contradictory guidelines. To mitigate these issues, we propose a novel synergistic strategy integrating retrieval-augmented generation (RAG) with direct preference optimization (DPO): RAG injects temporally accurate clinical knowledge, while DPO fine-tunes model preferences to enhance recognition and rejection of outdated or conflicting content. Experimental results demonstrate that our approach improves outdated recommendation rejection rate by 37% and achieves 89.2% consistency in resolving guideline conflicts—significantly outperforming standalone methods—and substantially enhances temporal coherence and clinical trustworthiness of LLMs in rapidly updating medical domains.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Models (LLMs) have great potential in the field of health care, yet they face great challenges in adapting to rapidly evolving medical knowledge. This can lead to outdated or contradictory treatment suggestions. This study investigated how LLMs respond to evolving clinical guidelines, focusing on concept drift and internal inconsistencies. We developed the DriftMedQA benchmark to simulate guideline evolution and assessed the temporal reliability of various LLMs. Our evaluation of seven state-of-the-art models across 4,290 scenarios demonstrated difficulties in rejecting outdated recommendations and frequently endorsing conflicting guidance. Additionally, we explored two mitigation strategies: Retrieval-Augmented Generation and preference fine-tuning via Direct Preference Optimization. While each method improved model performance, their combination led to the most consistent and reliable results. These findings underscore the need to improve LLM robustness to temporal shifts to ensure more dependable applications in clinical practice.

Problem

• Assessing medical knowledge drift in LLMs
• Mitigating conflicting treatment suggestions
• Improving temporal reliability in clinical guidelines

Innovation

• Developed DriftMedQA benchmark for guideline evolution
• Used Retrieval-Augmented Generation to enhance accuracy
• Applied Direct Preference Optimization for fine-tuning