Distilling Monocular Foundation Model for Fine-grained Depth Completion

2025-03-21

AI Summary

This paper addresses dense depth map prediction from sparse LiDAR inputs, particularly tackling the challenge of recovering fine geometric details in the absence of dense depth annotations. To this end, we propose a two-stage knowledge distillation framework. Methodologically: (1) we pioneer the use of monocular foundation models (e.g., MiDaS) to provide geometric priors and dense supervision; (2) we design a LiDAR-simulation data generation strategy to bridge the domain gap between monocular and sparse LiDAR modalities; and (3) we introduce a Scale- and Shift-Invariant (SSI) loss to explicitly model the inherent scale ambiguity of monocular depth estimation. The model is first self-supervised pre-trained on synthetic data and then fine-tuned on real-world sparse LiDAR inputs. Evaluated on the KITTI depth completion benchmark, our approach achieves state-of-the-art performance, ranking first.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Neural Architectures and Foundation Models
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Depth completion involves predicting dense depth maps from sparse LiDAR inputs. However, sparse depth annotations from sensors limit the availability of dense supervision, which is necessary for learning detailed geometric features. In this paper, we propose a two-stage knowledge distillation framework that leverages powerful monocular foundation models to provide dense supervision for depth completion. In the first stage, we introduce a pre-training strategy that generates diverse training data from natural images, which distills geometric knowledge to depth completion. Specifically, we simulate LiDAR scans by utilizing monocular depth and mesh reconstruction, thereby creating training data without requiring ground-truth depth. Besides, monocular depth estimation suffers from inherent scale ambiguity in real-world settings. To address this, in the second stage, we employ a scale- and shift-invariant loss (SSI Loss) to learn real-world scales when fine-tuning on real-world datasets. Our two-stage distillation framework enables depth completion models to harness the strengths of monocular foundation models. Experimental results demonstrate that models trained with our two-stage distillation framework achieve state-of-the-art performance, ranking extbf{first place} on the KITTI benchmark. Code is available at https://github.com/Sharpiless/DMD3C

Problem

• Predict dense depth maps from sparse LiDAR inputs
• Overcome sparse depth annotation limitations with dense supervision
• Resolve monocular depth scale ambiguity in real-world settings

Innovation

• Two-stage knowledge distillation framework
• Pre-training with monocular depth simulation
• Scale-invariant loss for real-world fine-tuning