BALTO: Balanced Token-Level Policy Optimization for Hallucination Mitigation

2026-06-14

AI Summary

This work addresses the challenge of hallucination in large language models on knowledge-intensive tasks, where existing reinforcement learning approaches using response-level rewards often mispenalize factually faithful content due to coarse granularity. The authors propose a token-level policy optimization framework that extracts verifiable factual claims from model outputs, aligns them with reference contexts, and maps claim-level verification outcomes into token-level reward labels to enable fine-grained credit assignment. By reallocating probability mass from unsupported to faithful content—rather than suppressing the entire response—the method enhances training stability and optimization efficiency. Evaluated on ConFiQA, RAGTruth, and FinLLM-Eval benchmarks, the approach achieves state-of-the-art Q-Scores across six model-benchmark combinations, significantly outperforming current post-training methods and striking a better balance between factual faithfulness and informativeness.

Abstract

Hallucinations remain a major obstacle to deploying large language models (LLMs) in knowledge-intensive settings, where generated responses must be faithfully grounded in provided evidence. Reinforcement learning (RL) is a promising direction for hallucination mitigation, but response-level faithfulness rewards suffer from a granularity mismatch: localized hallucinations can cause supported content to receive spurious penalties. Although recent work introduces fine-grained feedback such as claim-level verification and token-level rewards, unbalanced credit assignment can still induce length, verbosity, or optimization-noise biases. We propose BALTO, a Balanced Token-level Policy Optimization framework for hallucination mitigation. BALTO extracts checkable factual claims, verifies them against the reference context, and projects claim-level judgments to token-level labels. A balanced token-level credit assignment mechanism is introduced into the framework. This design redistributes probability mass from unsupported content toward faithful content, rather than suppressing the entire response. We systematically analyze the limitations of response-level rewards from a theoretical standpoint, and prove BALTO's advantages in training stability and optimization efficiency for hallucination mitigation. Experiments on ConFiQA, RAGTruth, and FinLLM-Eval show that BALTO achieves the highest faithfulness across all six model--benchmark settings and consistently outperforms existing post-training baselines in Q-Score, demonstrating a stronger faithfulness--informativeness trade-off.

Problem

• hallucination
• large language models
• faithfulness
• credit assignment
• reinforcement learning

Innovation

• token-level credit assignment
• hallucination mitigation
• balanced policy optimization
• fact grounding
• reinforcement learning