On-Policy Self-Alignment with Fine-grained Knowledge Feedback for Hallucination Mitigation

2024-06-18

AI Summary

To address hallucination in large language models (LLMs), this paper proposes RLFH—a reinforcement learning from human feedback framework that replaces human annotation with a “policy-as-judge” self-evaluation paradigm. Responses are decomposed into atomic facts, which are then assessed for factual accuracy and informativeness against external knowledge sources; these fine-grained evaluations are automatically mapped to token-level dense rewards, enabling online RL alignment without manual labeling. Key contributions include: (i) the first policy self-evaluation mechanism for LLMs; (ii) the first end-to-end mapping from sentence-level factual feedback to token-level reward signals; and (iii) overcoming limitations of offline fine-tuning and coarse-grained reward modeling. Evaluated on HotpotQA, SQuADv2, and Biography benchmarks, RLFH reduces hallucination rates significantly and improves factuality by 12.7% while preserving generation fluency and information completeness.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Hallucination occurs when large language models exhibit behavior that deviates from the boundaries of their knowledge during response generation. To address this critical issue, previous learning-based methods attempt to finetune models but are limited by off-policy sampling and coarse-grained feedback. In this paper, we present extit{{R}einforcement {L}earning {f}or {H}allucination} (RLFH), an on-policy self-alignment approach that enables LLMs to actively explore their knowledge boundaries and self-correct generation behavior through fine-grained feedback signals. RLFH introduces a self-assessment framework where the policy serves as its own judge. Through this framework, responses are automatically decomposed into atomic facts and their truthfulness and informativeness are assessed against external knowledge sources. The resulting fine-grained feedback at the statement level are then converted into token-level dense reward signals. This enables online reinforcement learning to achieve precise and timely optimization without human intervention. Comprehensive evaluations on HotpotQA, SQuADv2, and Biography benchmarks validate RLFH's effectiveness in hallucination mitigation.

Problem

• Mitigating hallucination in language models
• On-policy self-alignment with fine-grained feedback
• Reinforcement learning for precise optimization

Innovation

• On-policy self-alignment technique
• Fine-grained knowledge feedback
• Token-level dense reward signals