Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversity

2025-10-25

AI Summary

Multi-teacher knowledge distillation suffers from high computational overhead and deployment complexity. Method: This paper proposes a single-teacher, multi-perspective knowledge distillation framework: it constructs parallel branches within a single teacher model to explicitly model semantic-level viewpoint diversity. We innovatively design inter-angle diversity loss and intra-angle diversity loss to jointly optimize feature distributions—enforcing orthogonality across branches and consistency within each branch—without requiring auxiliary teacher networks to generate complementary knowledge views. Contribution/Results: Theoretical analysis shows our method reduces the upper bound of ensemble error. Extensive experiments demonstrate significant improvements over state-of-the-art knowledge-augmented distillation methods across diverse student architectures and benchmarks. Moreover, the framework is plug-and-play: it seamlessly integrates into mainstream distillation pipelines while enhancing generalization performance.

Technology Category

• Machine Learning: Multi-instance/Multi-view Learning
• Computer Vision: Multi-modal Vision
• Search and Optimization: Distributed Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Knowledge Distillation (KD) aims to train a lightweight student model by transferring knowledge from a large, high-capacity teacher. Recent studies have shown that leveraging diverse teacher perspectives can significantly improve distillation performance; however, achieving such diversity typically requires multiple teacher networks, leading to high computational costs. In this work, we propose a novel cost-efficient knowledge augmentation method for KD that generates diverse multi-views by attaching multiple branches to a single teacher. To ensure meaningful semantic variation across multi-views, we introduce two angular diversity objectives: 1) constrained inter-angle diversify loss, which maximizes angles between augmented views while preserving proximity to the original teacher output, and 2) intra-angle diversify loss, which encourages an even distribution of views around the original output. The ensembled knowledge from these angularly diverse views, along with the original teacher, is distilled into the student. We further theoretically demonstrate that our objectives increase the diversity among ensemble members and thereby reduce the upper bound of the ensemble's expected loss, leading to more effective distillation. Experimental results show that our method surpasses an existing knowledge augmentation method across diverse configurations. Moreover, the proposed method is compatible with other KD frameworks in a plug-and-play fashion, providing consistent improvements in generalization performance.

Problem

• Enhancing knowledge distillation through single-teacher multi-view augmentation
• Generating angularly diverse views to reduce ensemble loss bounds
• Improving student model performance without multiple teacher networks

Innovation

• Attaching multiple branches to single teacher
• Using angular diversity objectives for view variation
• Ensembling diverse views for improved distillation