GATES: Self-Distillation under Privileged Context with Consensus Gating

2026-02-24

AI Summary

This work addresses the challenge of self-distillation in document-grounded question answering under an asymmetric setting where documents are available during training but absent at test time, and no ground-truth labels or external supervision exist. The authors propose a consensus-gated self-distillation approach based on multi-trajectory sampling: a teacher model generates multiple reasoning trajectories, and their consensus serves as a dynamic reliability signal to modulate knowledge distillation. Crucially, the full reasoning trajectories—not just final answers—are used to train the student model. This method extracts stable, dense supervision signals from online-generated trajectories without assuming the teacher is always correct. Experiments show that it improves accuracy from 46.0% to 62.0% in in-domain asymmetric evaluation and boosts maj@8 accuracy from 20.2% to 35.4% on a document-free mathematical reasoning benchmark.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

We study self-distillation in settings where supervision is unreliable: there are no ground truth labels, verifiable rewards, or external graders to evaluate answers. We focus on document-grounded question answering with asymmetric context, where a single model serves as both tutor (with access to a relevant source document during training) and student (answering from the question alone at test time). Rather than assuming tutor correctness, we derive supervision online from tutor consensus by sampling multiple document-grounded reasoning traces and using agreement to gate learning. Conditioned on this reliability signal, we distill knowledge through full tutor reasoning trajectories (not just final answers), providing a dense and stable learning signal. Empirically, this consensus-gated trajectory distillation substantially improves transfer to the document-free student. Held-out in-domain accuracy under asymmetric evaluation improves from 46.0\% to 62.0\%, and average (maj@8) accuracy on public document-free math benchmarks improves from 20.2\% to 35.4\%.

Problem

• self-distillation
• unreliable supervision
• document-grounded QA
• asymmetric context
• consensus gating

Innovation

• self-distillation
• consensus gating
• trajectory distillation
• asymmetric context
• document-grounded QA