xModel-KD: Cross-modal Knowledge Distillation for 3D Scene Perception using LiDAR

2026-05-28

AI Summary

This work addresses the high annotation cost and limitations of single-modality perception in 3D point cloud segmentation by proposing a cross-modal knowledge distillation framework that effectively integrates 2D image texture cues with 3D geometric structures to learn unified per-point representations. Leveraging pre-trained 2D and 3D backbone networks, a cross-modal fusion encoder, and a multi-view contrastive alignment objective, the method enables efficient dense prediction without requiring extensive labeled data. Experimental results demonstrate that the approach achieves an absolute improvement of 2% in mean Intersection over Union (mIoU) over LiDAR-only baselines, validating its effectiveness and scalability for 3D scene understanding.

Abstract

Point cloud segmentation is a fundamental task in 3D scene understanding. Its progress is constrained by the high cost and time required for dense 3D annotations, making labeled samples difficult to obtain. Beyond annotation scarcity, different sensing modalities face inherent limitations. 2D images provide rich texture and appearance cues, yet they lack explicit depth and geometric structure. In contrast, 3D point clouds capture accurate spatial geometry but are sparse and contain no texture information. As a result, relying on a single modality restricts the richness of learned representations and weakens generalization. Although recent multi-modal methods that combine 3D point clouds with 2D images have demonstrated strong performance in tasks such as classification and retrieval, they typically depend on large-scale labeled datasets and have not been fully exploited for data-efficient dense prediction. To address these limitations, we propose a novel cross-modal knowledge distillation framework, xModel-KD, for 3D point cloud segmentation. Our method exploits the complementary strengths of 2D texture and 3D geometry by learning unified per-point representations through cross-modal alignment. Specifically, we design a cross-modal fusion encoder trained with a contrastive objective that enforces feature consistency between corresponding 2D and 3D representations across multiple views. By integrating powerful pre-trained backbones with a targeted fusion strategy, the proposed framework effectively transfers appearance cues from images to geometry-aware point features. Experimental results show that cross-modal fusion achieves a 2% absolute improvement in mIoU over a LiDAR-only baseline, demonstrating the benefit of leveraging complementary multi-modal information for scalable and annotation-efficient 3D scene understanding.

Problem

• 3D point cloud segmentation
• cross-modal learning
• annotation scarcity
• multi-modal fusion
• dense prediction

Innovation

• cross-modal knowledge distillation
• 3D point cloud segmentation
• multi-modal fusion
• contrastive learning
• LiDAR