GhostPoint: Self-Supervised Representation Learning by Hallucinating Occluded LiDAR Structure

2026-08-14

AI Summary

This study addresses the insufficient robustness to occluded structures in LiDAR self-supervised learning caused by visible surface bias. We propose GhostPoint, a novel framework that introduces an instance voxel dilation-based neighborhood hallucination mechanism and a predictor-level dual supervision strategy. Combined with an encoder-predictor dual-branch architecture and teacher model distillation, this approach explicitly models geometry beyond observed surfaces, overcoming the limitations of relying solely on visible data. GhostPoint achieves state-of-the-art performance on the nuScenes and Waymo datasets, significantly improving 3D detection in sparse scanning and few-label scenarios. Consequently, the proposed method effectively enhances model perception capabilities for occluded objects, establishing a new benchmark for robust representation learning in autonomous driving applications.

Abstract

3D object detection from LiDAR point clouds is a core problem in autonomous driving. Recent advances in self-supervised learning (SSL) enable scalable pretraining and transfers well to per-point tasks such as semantic and panoptic segmentation, but transfer to 3D detection remains weaker. We analyze recent SSL methods and find that most objectives are defined only on measured LiDAR returns from visible surfaces, leaving occluded and unobserved regions unconstrained. This visible-surface bias can be sufficient for point-wise prediction, but 3D detection requires robustness to missing structure. To address this gap, we propose GhostPoint, an SSL framework that hallucinates latent features in local neighborhoods around discovered instances, generated via a novel instance voxel dilation. In GhostPoint, an encoder processes observed returns, and an additional predictor infers neighborhood representations from observed context. In addition to standard encoder-level supervision, we introduce a predictor-level supervision scheme on sampled voxels from generated neighborhoods. Specifically, observed (visible/masked) voxels match teacher-encoder targets, while unobserved voxels match teacher-predictor hallucinations. This design encourages the learned representation to explicitly model structure beyond observed returns. Extensive evaluations on nuScenes and Waymo demonstrate that our method achieves state-of-the-art performance, consistently improving downstream 3D detection, especially under sparse scans and limited labels.

Problem

• Self-Supervised Learning
• 3D Object Detection
• LiDAR Point Clouds
• Occlusion
• Transfer Learning

Innovation

• Self-Supervised Learning
• Occluded LiDAR Structure
• Instance Voxel Dilation
• Predictor-Level Supervision
• 3D Object Detection