Contrastive Representation Distillation via Multi-Scale Feature Decoupling

2025-02-09

AI Summary

Existing knowledge distillation methods suffer from insufficient local feature disentanglement and overreliance on global representations, leading to suboptimal utilization of teacher knowledge. To address this, we propose Multi-Scale Feature Disentanglement (MSFD), a novel distillation framework that orthogonally decomposes teacher features along both spatial granularity and semantic hierarchy, thereby constructing scale-aware local feature subspaces. MSFD further introduces intra-batch contrastive learning to align features across scales. Crucially, it achieves enhanced feature transfer efficiency without increasing student model parameters. Extensive experiments on CIFAR-100 and ImageNet demonstrate that lightweight students—including ResNet-18 and MobileNetV2—surpass their larger teacher counterparts (e.g., ResNet-50 and ResNet-101) in accuracy. These results validate the effectiveness and generalizability of MSFD’s local–global collaborative distillation paradigm.

Technology Category

• Machine Learning: Multi-instance/Multi-view Learning
• Computer Vision: Representation Learning for Vision
• Search and Optimization: Learning to Search

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies

Abstract

Knowledge distillation is a technique aimed at enhancing the performance of a smaller student network without increasing its parameter size by transferring knowledge from a larger, pre-trained teacher network. Previous approaches have predominantly focused on distilling global feature information while overlooking the importance of disentangling the diverse types of information embedded within different regions of the feature. In this work, we introduce multi-scale decoupling in the feature transfer process for the first time, where the decoupled local features are individually processed and integrated with contrastive learning. Moreover, compared to previous contrastive learning-based distillation methods, our approach not only reduces computational costs but also enhances efficiency, enabling performance improvements for the student network using only single-batch samples. Extensive evaluations on CIFAR-100 and ImageNet demonstrate our method's superiority, with some student networks distilled using our method even surpassing the performance of their pre-trained teacher networks. These results underscore the effectiveness of our approach in enabling student networks to thoroughly absorb knowledge from teacher networks.

Problem

• Enhance student network performance
• Multi-scale feature decoupling
• Reduce computational costs and enhance efficiency

Innovation

• Multi-scale feature decoupling
• Contrastive learning integration
• Single-batch sample efficiency