Synthetic-to-Real Self-supervised Robust Depth Estimation via Learning with Motion and Structure Priors

2025-03-26

AI Summary

To address the robustness degradation of monocular self-supervised depth estimation in complex outdoor scenarios—such as day-night transitions and adverse weather (e.g., rain, fog)—caused by the absence of ground-truth depth annotations and domain shift, this paper proposes the first synthetic-to-real self-supervised depth estimation framework. Our method operates without real-depth supervision and freezes a pre-trained daytime model while performing synthetic adverse-weather training. Key contributions include: (1) a motion-structure prior transfer mechanism based on cost volume; (2) weather-invariant region identification and consistency-aware reweighting of pseudo-labels; and (3) explicit depth distribution regularization to mitigate synthetic-to-real domain discrepancy. Extensive experiments demonstrate state-of-the-art performance: on nuScenes and RobotCar (covering day/night/rain conditions), our approach achieves average improvements of 7.5% in AbsRel and 4.3% in RMSE; on DrivingStereo (rain/fog), it exhibits superior zero-shot generalization over prior art.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Intelligent Robots: State Estimation
• Machine Learning: Adversarial Learning & Robustness

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Self-supervised depth estimation from monocular cameras in diverse outdoor conditions, such as daytime, rain, and nighttime, is challenging due to the difficulty of learning universal representations and the severe lack of labeled real-world adverse data. Previous methods either rely on synthetic inputs and pseudo-depth labels or directly apply daytime strategies to adverse conditions, resulting in suboptimal results. In this paper, we present the first synthetic-to-real robust depth estimation framework, incorporating motion and structure priors to capture real-world knowledge effectively. In the synthetic adaptation, we transfer motion-structure knowledge inside cost volumes for better robust representation, using a frozen daytime model to train a depth estimator in synthetic adverse conditions. In the innovative real adaptation, which targets to fix synthetic-real gaps, models trained earlier identify the weather-insensitive regions with a designed consistency-reweighting strategy to emphasize valid pseudo-labels. We introduce a new regularization by gathering explicit depth distributions to constrain the model when facing real-world data. Experiments show that our method outperforms the state-of-the-art across diverse conditions in multi-frame and single-frame evaluations. We achieve improvements of 7.5% and 4.3% in AbsRel and RMSE on average for nuScenes and Robotcar datasets (daytime, nighttime, rain). In zero-shot evaluation of DrivingStereo (rain, fog), our method generalizes better than the previous ones.

Problem

• Robust depth estimation in diverse outdoor conditions
• Bridging synthetic-to-real gap with motion-structure priors
• Improving accuracy in adverse weather scenarios

Innovation

• Uses synthetic-to-real framework with motion-structure priors
• Trains depth estimator with frozen daytime model
• Applies consistency-reweighting for weather-insensitive regions