Feasibility of Indoor Frame-Wise Lidar Semantic Segmentation via Distillation from Visual Foundation Model

2026-04-20

AI Summary

Indoor LiDAR frame-level semantic segmentation remains challenging due to the scarcity of annotated data for training deep models. This work proposes a novel, annotation-free cross-modal knowledge distillation approach that transfers semantic knowledge from vision foundation models to LiDAR frames by precisely aligning image-derived semantics with 3D point clouds. It is the first to demonstrate the feasibility of leveraging vision foundation models for frame-level semantic distillation on indoor LiDAR data and introduces the first small-scale, manually annotated indoor LiDAR semantic dataset for evaluation. Experimental results show that the method achieves 56% mIoU using pseudo-labels and 36% mIoU on ground-truth annotations, highlighting its effectiveness and potential for label-efficient semantic understanding in indoor environments.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Frame-wise semantic segmentation of indoor lidar scans is a fundamental step toward higher-level 3D scene understanding and mapping applications. However, acquiring frame-wise ground truth for training deep learning models is costly and time-consuming. This challenge is largely addressed, for imagery, by Visual Foundation Models (VFMs) which segment image frames. The same VFMs may be used to train a lidar scan frame segmentation model via a 2D-to-3D distillation pipeline. The success of such distillation has been shown for autonomous driving scenes, but not yet for indoor scenes. Here, we study the feasibility of repeating this success for indoor scenes, in a frame-wise distillation manner by coupling each lidar scan with a VFM-processed camera image. The evaluation is done using indoor SLAM datasets, where pseudo-labels are used for downstream evaluation. Also, a small manually annotated lidar dataset is provided for validation, as there are no other lidar frame-wise indoor datasets with semantics. Results show that the distilled model achieves up to 56% mIoU under pseudo-label evaluation and around 36% mIoU with real-label, demonstrating the feasibility of cross-modal distillation for indoor lidar semantic segmentation without manual annotations.

Problem

• indoor LiDAR
• semantic segmentation
• frame-wise
• annotation scarcity
• cross-modal distillation

Innovation

• cross-modal distillation
• lidar semantic segmentation
• visual foundation model
• indoor scene understanding
• pseudo-labeling