Hallucination Self-Play: Bootstrapping Reinforced Detector via Evolved Generator

2026-07-08

AI Summary

This work addresses the scarcity of high-quality annotated data for hallucination detection in large language models by proposing a self-play co-evolution framework. Starting from a small set of human-annotated samples, an initial detector serves as a reward model to guide a generator—via reinforcement learning augmented with rule-based constraints and AI feedback—in producing increasingly deceptive hallucinated examples. This process enables iterative, mutual refinement of both generator and detector, achieving dynamic co-evolution without reliance on external supervision and overcoming the limitations of static synthetic data. Experimental results demonstrate that a small LLM trained within this framework attains hallucination detection performance on the RAGTruth benchmark that matches or even surpasses that of state-of-the-art larger models.

Abstract

Identifying faithfulness hallucinations in LLM-generated outputs remains challenging due to the scarcity of high-quality annotated data. Recent work relies on advanced LLMs to synthesize training data, including rationales, labels, and hallucinated claims. However, these methods treat the generator as a static component, limiting iterative improvement of the detector. To address this limitation, we introduce Hallucination Self-Play (HSP), a novel framework that enables the detector to bootstrap with an evolved generator. HSP involves two roles initialized from the same base model, a detector that assesses the faithfulness of model outputs, and a generator that produces increasingly hard-to-detect hallucinated responses. Specifically, the detector is first fine-tuned on human-labeled data and then employed as a reward model to train the generator via reinforcement learning from AI feedback (RLAIF). In turn, the evolved generator synthesizes hallucination data to further optimize the detector through rule-based reinforcement learning. Experiments on RAGTruth benchmark and two model families demonstrate that the proposed framework can progressively enhance a small LLM to match or even outperform advanced LLMs without external supervision. Our code is available at https://anonymous.4open.science/r/Hallucination-Self-Play-50B5 .

Problem

• hallucination
• faithfulness
• LLM
• data scarcity
• detector

Innovation

• Hallucination Self-Play
• reinforcement learning from AI feedback
• faithfulness detection
• evolved generator
• self-bootstrapping