Winning Big with Small Models: Knowledge Distillation vs. Self-Training for Reducing Hallucination in QA Agents

2025-02-26

AI Summary

Large language models (LLMs) exhibit severe hallucination in customer support applications, while reliance on proprietary models and human annotation incurs high operational costs. Method: Targeting Samsung smart TV user manuals, this work develops a lightweight retrieval-augmented generation (RAG) system. First, self-training is shown to suppress hallucination as effectively as knowledge distillation but at lower cost. Second, a contextualized “I don’t know” response mechanism is introduced, significantly improving robustness to unanswerable questions and retrieval failures. Third, LLM-synthesized training data is empirically validated to surpass human-annotated data in quality, enabling full open-source pipeline adoption. Contribution/Results: Leveraging only open-weight small language models (e.g., Phi-3, Llama-3) and synthetic data, the system achieves substantially reduced hallucination rates and improved accuracy in identifying unanswerable queries—delivering a scalable, low-cost, low-dependency, and low-hallucination QA solution.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Question Answering
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

The deployment of Large Language Models (LLMs) in customer support is constrained by hallucination-generating false information-and the high cost of proprietary models. To address these challenges, we propose a retrieval-augmented question-answering (QA) pipeline and explore how to balance human input and automation. Using a dataset of questions about a Samsung Smart TV user manual, we demonstrate that synthetic data generated by LLMs outperforms crowdsourced data in reducing hallucination in finetuned models. We also compare self-training (fine-tuning models on their own outputs) and knowledge distillation (fine-tuning on stronger models' outputs, e.g., GPT-4o), and find that self-training achieves comparable hallucination reduction. We conjecture that this surprising finding can be attributed to increased exposure bias issues in the knowledge distillation case and support this conjecture with post hoc analysis. We also improve robustness to unanswerable questions and retrieval failures with contextualized"I don't know"responses. These findings show that scalable, cost-efficient QA systems can be built using synthetic data and self-training with open-source models, reducing reliance on proprietary tools or costly human annotations.

Problem

• Reduces hallucination in QA agents
• Balances human input and automation
• Improves robustness to unanswerable questions

Innovation

• Retrieval-augmented QA pipeline
• Self-training reduces hallucination
• Synthetic data outperforms crowdsourced