HilDA: Hierarchical Distillation with Diffusion for Advancing Self-Supervised LiDAR Pre-trainin

2026-06-18

AI Summary

This work addresses the scarcity of LiDAR annotations in autonomous driving and the limited effectiveness of existing vision foundation model distillation approaches in leveraging the teacher model’s semantic structure and temporal information. To this end, the authors propose a self-supervised LiDAR pretraining framework that, for the first time, integrates hierarchical vision–LiDAR distillation with a diffusion mechanism to jointly model semantic “what” and geometric “where” without requiring labels. The method enhances semantic consistency and spatiotemporal coherence through multi-level feature alignment, global context distillation, and a temporally consistent occupancy diffusion objective. Experiments demonstrate that the proposed approach achieves state-of-the-art performance on cross-modal distillation benchmarks and significantly outperforms existing distillation methods in 3D object detection, scene flow estimation, and semantic occupancy prediction tasks.

Abstract

Leveraging Vision Foundation Models (VFMs) for camera-to-LiDAR knowledge distillation offers a promising solution to the scarcity of annotated data needed to represent the immense geometric and kinematic diversity of real-world autonomous driving (AD). However, current approaches typically treat VFMs as black-box teachers, relying exclusively on frame-wise feature similarity. Consequently, they do not fully exploit the teacher's layer-wise semantic structure and global context, as well as the rich spatiotemporal information inherent in LiDAR sequences. We propose HilDA, a self-supervised pretraining framework for LiDAR backbones that better captures the semantic what and geometric where needed for driving tasks. HilDA combines hierarchical distillation comprising multi-layer distillation for progressive semantic alignment and global context distillation for scene-level semantics, with a temporal occupancy diffusion objective promoting spatiotemporal consistency. Models pre-trained with HilDA achieve state-of-the-art results on cross-modal distillation benchmarks and outperform models trained via prior distillation approaches on 3D object detection, scene flow, and semantic occupancy prediction. Code available at: https://maxiuw.github.io/hilda.

Problem

• self-supervised learning
• LiDAR pre-training
• knowledge distillation
• spatiotemporal consistency
• autonomous driving

Innovation

• Hierarchical Distillation
• Diffusion Model
• Self-Supervised Pretraining
• LiDAR Representation Learning
• Cross-Modal Knowledge Distillation