Learning from Consensus and Disagreement: Unsupervised On-Policy Self-Distillation with Minority-Trajectory Contrast

2026-08-09

AI Summary

This work addresses the limitation of existing policy-based self-distillation methods, which rely on external supervision and struggle to enhance language models’ reasoning capabilities under fully unsupervised conditions. To overcome this, we propose CoDA, a novel framework that, for the first time, constructs reliable privileged information in an unsupervised setting by leveraging consensus among model-generated reasoning trajectories as positive guidance and minority trajectories as negative samples for contrastive calibration. CoDA further integrates a frozen self-teacher, answer-level consensus identification, and a KTO-style reference anchoring mechanism to effectively mitigate error amplification caused by incorrect consensus. Experimental results demonstrate that CoDA significantly outperforms self-generation baselines on competitive mathematical reasoning benchmarks, simultaneously improving both reasoning performance and training stability.

Abstract

On-policy self-distillation improves language-model reasoning by querying a teacher on states actually visited by the student. Recent methods create a powerful information asymmetry by exposing the teacher to privileged context, yet they fundamentally rely on external supervision---such as gold solutions or verifiers---to construct this advantage. We introduce CoDA (Consensus and Disagreement Alignment), a fully unsupervised framework that creates reliable privileged information entirely from the latent uncertainty structure of a model's own unlabeled rollouts. CoDA extracts two complementary signals. In the positive branch, answer-level consensus identifies a stable reasoning mode, which conditions a frozen self-teacher to provide dense distributional guidance on fresh student trajectories. However, because agreement does not guarantee correctness, positive-only distillation risks amplifying correlated errors into a false consensus. To break this harmful feedback loop, CoDA incorporates a negative branch that exploits disagreement: minority trajectories are treated as unstable alternatives and gently penalized via a reference-anchored, KTO-style calibration objective. This unpaired binary feedback provides robust regularization without requiring the strong assumption that the consensus is the absolute ground truth. Empirical evaluations on competition-level mathematical benchmarks demonstrate that CoDA significantly improves reasoning, outperforming self-generated baselines and effectively stabilizing training against erroneous consensus.

Problem

• unsupervised
• self-distillation
• reasoning
• consensus
• disagreement

Innovation

• unsupervised self-distillation
• on-policy learning
• consensus-disagreement alignment
• minority-trajectory contrast
• reasoning stabilization