Consensus as Privileged Context for Label-Free Self-Distillation

2026-07-15

AI Summary

This work addresses the challenge of leveraging consensus signals from large language models to improve reasoning accuracy in the absence of labeled data. The authors propose CANON, a novel method that, for the first time, transforms majority answers from multi-path reasoning into dense token-level supervision signals rather than merely using them for filtering or scalar rewards. CANON employs a frozen consensus-anchored teacher model to perform unlabeled self-distillation on the base model and introduces a consensus-guided conditional generation mechanism. Experimental results demonstrate that CANON achieves up to a 12-percentage-point improvement in pass@1 accuracy on mathematical and scientific reasoning tasks, outperforms unsupervised reinforcement learning by 6 points at roughly one-seventh the computational cost, approaches the performance of teacher models trained with gold labels, and successfully solves problems on which the original model failed across all 32 attempts, substantially enhancing majority-vote accuracy.

Abstract

Sampling multiple solutions and returning the majority answer is among the most reliable ways to improve the reasoning accuracy of large language models without labels, and a growing family of methods converts this consensus signal into training supervision. However, existing approaches use consensus only in restricted forms: as a filter that selects solutions for fine-tuning, as a preference between answers, or as a scalar reward for reinforcement learning, discarding most of the information that the agreeing solutions contain. We present CANON (Consensus-ANchored self-distillatiON), a label-free training method that turns consensus into dense, token-level supervision. For each unlabeled prompt, CANON samples multiple solutions, extracts the majority answer, and conditions a frozen snapshot of the model on a solution that reaches it; this consensus-anchored teacher then supervises the model on its own rollouts at every token. Experiments on mathematical and scientific reasoning benchmarks show that CANON improves pass@1 by up to 12 points, outperforming label-free reinforcement learning by 6 points at a seventh of its compute and approaching a teacher conditioned on gold solutions; trained on pooled unlabeled data, it transfers to held-out benchmarks, matching training methods that use gold labels. Analysis suggests that the improvements are not pure distribution sharpening: after training, the model solves problems it previously never solved in 32 attempts, and its majority vote itself becomes more accurate.

Problem

• consensus
• label-free
• self-distillation
• large language models
• reasoning accuracy

Innovation

• consensus-based supervision
• label-free self-distillation
• token-level distillation
• majority voting
• unlabeled data training