Fine-grained Image-to-LiDAR Contrastive Distillation with Visual Foundation Models

2024-05-23

arXiv.org

AI Summary

To address semantic misalignment—e.g., similar objects incorrectly classified as distinct categories—and insufficient robustness of 3D representations in image-LiDAR contrastive distillation, this paper proposes a weakly supervised fine-grained contrastive distillation framework. Methodologically, it (1) leverages vision foundation models (SAM/DINO) to generate pixel-level semantic labels, enabling point-to-pixel weak supervision without LiDAR annotations; (2) imposes a von Mises–Fisher (vMF) distribution constraint to enforce semantic directional consistency and mitigate inter-modal feature distribution shift; and (3) introduces a spatial- and class-adaptive point sampling strategy to enhance representation quality in discriminative regions. Evaluated on multiple 3D detection and segmentation benchmarks, the framework significantly outperforms existing contrastive distillation methods, achieving substantial gains in semantic consistency and cross-modal generalization.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Multimodal Learning
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Contrastive image-to-LiDAR knowledge transfer, commonly used for learning 3D representations with synchronized images and point clouds, often faces a self-conflict dilemma. This issue arises as contrastive losses unintentionally dissociate features of unmatched points and pixels that share semantic labels, compromising the integrity of learned representations. To overcome this, we harness Visual Foundation Models (VFMs), which have revolutionized the acquisition of pixel-level semantics, to enhance 3D representation learning. Specifically, we utilize off-the-shelf VFMs to generate semantic labels for weakly-supervised pixel-to-point contrastive distillation. Additionally, we employ von Mises-Fisher distributions to structure the feature space, ensuring semantic embeddings within the same class remain consistent across varying inputs. Furthermore, we adapt sampling probabilities of points to address imbalances in spatial distribution and category frequency, promoting comprehensive and balanced learning. Extensive experiments demonstrate that our approach mitigates the challenges posed by traditional methods and consistently surpasses existing image-to-LiDAR contrastive distillation methods in downstream tasks. The source code is available at https://github.com/Eaphan/OLIVINE.

Problem

• Contrastive Learning
• LiDAR Data Conversion
• 3D Spatial Understanding

Innovation

• Contrastive Learning
• Visual Foundation Model
• Multi-task Performance Enhancement