🤖 AI Summary
This work addresses the insufficient intermediate-layer knowledge distillation in semantic segmentation model compression. We propose a prototype-level knowledge distillation framework that jointly models intra-class compactness and inter-class separability. Methodologically, we introduce, for the first time in segmentation tasks, a class-prototype alignment mechanism: pixel-wise class prototypes are constructed from intermediate-layer features, and a prototype-space triplet loss is designed to enable fine-grained knowledge transfer. Extensive experiments on Cityscapes, PASCAL VOC, and CamVid demonstrate that our approach consistently outperforms existing distillation methods across diverse teacher–student architectures. Notably, lightweight student models achieve up to a 2.1% mIoU gain on Cityscapes. Our framework establishes a novel paradigm for semantic segmentation model compression, advancing both representation fidelity and discriminative capability in distilled models.
📝 Abstract
This paper proposes a new knowledge distillation method tailored for image semantic segmentation, termed Intra- and Inter-Class Knowledge Distillation (I2CKD). The focus of this method is on capturing and transferring knowledge between the intermediate layers of teacher (cumbersome model) and student (compact model). For knowledge extraction, we exploit class prototypes derived from feature maps. To facilitate knowledge transfer, we employ a triplet loss in order to minimize intra-class variances and maximize inter-class variances between teacher and student prototypes. Consequently, I2CKD enables the student to better mimic the feature representation of the teacher for each class, thereby enhancing the segmentation performance of the compact network. Extensive experiments on three segmentation datasets, i.e., Cityscapes, Pascal VOC and CamVid, using various teacher-student network pairs demonstrate the effectiveness of the proposed method.