Resolving Knowledge Conflicts in Domain-specific Data Selection: A Case Study on Medical Instruction-tuning

2025-05-28

AI Summary

To address hallucination and capability degradation in medical instruction tuning—caused by conflicts between pretrained knowledge and instruction-contextual knowledge—this paper proposes Knowledge-Aware Data Selection (KDS), a novel framework for instruction dataset curation. KDS is the first to systematically model and quantify such knowledge conflicts, introducing an interpretable evaluation metric grounded in two dimensions: “context-memory alignment” and “intra-memory consistency.” Leveraging this metric, KDS implements conflict-aware filtering and high-quality, diverse sampling strategies, culminating in a specialized medical instruction dataset. Extensive evaluation across multiple large language models in healthcare demonstrates that KDS consistently enhances instruction-following capability and generalization while substantially mitigating hallucination, yielding average performance gains of 4.2–7.8 percentage points.

Technology Category

• Data Mining & Knowledge Management: Knowledge Acquisition from the Web
• Knowledge Representation and Reasoning: Knowledge Acquisition
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Domain-specific instruction-tuning has become the defacto standard for improving the performance of large language models (LLMs) in specialized applications, e.g., medical question answering. Since the instruction-tuning dataset might contain redundant or low-quality data, data selection (DS) is usually required to maximize the data efficiency. Despite the successes in the general domain, current DS methods often struggle to select the desired data for domain-specific instruction-tuning. One of the main reasons is that they neglect the impact of knowledge conflicts, i.e., the discrepancy between LLMs' pretrained knowledge and context knowledge of instruction data, which could damage LLMs' prior abilities and lead to hallucination. To this end, we propose a simple-yet-effective Knowledge-aware Data Selection (namely KDS) framework to select the domain-specific instruction-tuning data that meets LLMs' actual needs. The core of KDS is to leverage two knowledge-aware metrics for quantitatively measuring knowledge conflicts from two aspects: context-memory knowledge alignment and intra-memory knowledge consistency. By filtering the data with large knowledge conflicts and sampling the high-quality and diverse data, KDS can effectively stimulate the LLMs' abilities and achieve better domain-specific performance. Taking the medical domain as the testbed, we conduct extensive experiments and empirically prove that KDS surpasses the other baselines and brings significant and consistent performance gains among all LLMs. More encouragingly, KDS effectively improves the model generalization and alleviates the hallucination problem.

Problem

• Addressing knowledge conflicts in domain-specific data selection for LLMs
• Improving medical instruction-tuning by filtering high-conflict, low-quality data
• Enhancing LLM performance and reducing hallucination via knowledge-aware metrics

Innovation

• Knowledge-aware Data Selection (KDS) framework
• Quantify knowledge conflicts with two metrics
• Filter and sample high-quality diverse data