Distill Where the Student Goes: Teacher-Regularized RL for English-Evidence Cross-Lingual RAG

2026-07-03

AI Summary

In cross-lingual retrieval-augmented generation (RAG), reliance on English evidence often induces language drift and unreliable evidence utilization, degrading the quality of non-English outputs. To address this, this work proposes TR-RAG, a method integrating reward decomposition with prefix-aware online distillation. TR-RAG applies prefix-level reverse KL regularization via a frozen teacher model and decomposes the reward signal into three components: language consistency, character 3-gram recall, and LLM-judged evidence correctness. Evaluated on three multilingual RAG benchmarks, TR-RAG substantially improves overall performance, prevents catastrophic language inconsistency—yielding gains of up to 27 percentage points—and in some cases surpasses the 70B-parameter teacher model in character 3-gram recall.

Abstract

Cross-lingual retrieval-augmented generation (RAG) is often deployed in an English-evidence regime, where users query in diverse languages but retrieved passages remain English. In this setting, generation can fail despite strong base models: English evidence induces language drift (English or code-switching outputs) and models use evidence unreliably when producing non-English answers. We attribute these failures to two post-training challenges: (i) errors are prefix-dependent, so fixed-trajectory supervision suffers from prefix mismatch; and (ii) sequence-level (partly discrete / judge-based) rewards yield noisy credit assignment and high-variance updates. We propose TR-RAG, a teacher-regularized RL recipe that couples reward optimization with on-policy distillation on student-visited prefixes. A compact student samples on-policy answers, while a stronger frozen teacher is queried only on those prefixes and provides a prefix-wise student-to-teacher reverse-KL anchor. We further introduce a reward decomposition for English-evidence multilingual generation, combining language consistency, character 3-gram recall, and an LLM-judge score for evidence-grounded correctness. Across three benchmarks -- BioASQ-ENKB5, Hotpot-ENKB5, and naturally multilingual MKQA -- and two backbones, TR-RAG improves the composite of language adherence and evidence-grounded correctness over strong baselines. Crucially, the teacher anchor acts as a safety net: on in-domain languages it prevents the large language-consistency collapses (up to ~27 percentage points) that reward-only RL can suffer by drifting below even the base model, while on distant out-of-distribution languages -- where reward-only RL stalls at the base model's ceiling -- it still improves evidence grounding; and on character 3-gram recall the compact student sometimes surpasses its 70B teacher.

Problem

• cross-lingual RAG
• language drift
• evidence grounding
• English-evidence regime
• multilingual generation

Innovation

• teacher-regularized RL
• cross-lingual RAG
• on-policy distillation
• reward decomposition
• language drift mitigation