Heterogeneous and Adept Snapshot Distillation for 3D Semantic Segmentation

2026-06-23

AI Summary

This work proposes a novel framework that enhances 3D semantic segmentation performance of point cloud–only models without incurring additional inference overhead. By integrating Information-guided Heterogeneous Distillation (IHD) with Ability-aware Snapshot Distillation (ASD), the method leverages a multimodal teacher model and snapshot expert models generated during training to enable efficient, class-specific knowledge transfer—each expert focusing on its most proficient categories. The approach employs information-guided filtering and heterogeneous knowledge distillation to effectively fuse multi-source supervisory signals, all without requiring extra inputs or substantial computational cost. Evaluated on ScanNetV2 and S3DIS benchmarks, the framework achieves state-of-the-art performance and can be seamlessly integrated into existing 3D segmentation architectures to significantly boost accuracy.

Abstract

Multi-modal fusion and multi-model ensembling are prevalent in enhancing the performance of 3D semantic segmentation. Despite the impressive performance, these methods either rely on auxiliary input signals or suffer from costly computational expense. To efficaciously enhance the segmentation performance without introducing intolerable costs, we propose to transfer the rich knowledge from the multi-modal model (i.e., point clouds and images) and multiple model experts to the point-cloudbased network through knowledge distillation. Specifically, we present Information-oriented Heterogeneous Distillation (IHD) to help the uni-modal model absorb the complementary knowledge from the multi-modal teacher. We design the Information-Oriented Filtering (IOF) strategy to select informative images from the continuous image sequence for multi-modal fusion. This practice can boost the performance of the multi-modal teacher, thus benefiting the learning of the student. Besides, as opposed to vanilla model ensembling that requires the separate training of each expert, we propose Adept Snapshot Distillation (ASD). ASD treats the freely available model snapshots generated during the training phase as multiple experts, which significantly reduces the training cost for model ensembling. For each expert teacher, it only provides supervision to the student in the class where it is adept. The resulting Heterogeneous and Adept Snapshot Knowledge Distillation, dubbed HAS-KD, attains state-of-the-art results on ScanNetV2 and S3DIS datasets. HAS-KD can be seamlessly integrated into contemporary 3D segmentation algorithms and bring considerable gains without introducing extra inference burdens. The code will be made publicly available upon publication.

Problem

• 3D semantic segmentation
• knowledge distillation
• multi-modal fusion
• model ensembling
• computational cost

Innovation

• knowledge distillation
• 3D semantic segmentation
• multi-modal fusion
• model ensembling
• snapshot distillation