Re-Depth Anything: Test-Time Depth Refinement via Self-Supervised Re-lighting

2025-12-19

AI Summary

Monocular depth estimation suffers from poor out-of-distribution generalization in real-world scenes, with state-of-the-art models (e.g., Depth Anything V2) exhibiting significant accuracy degradation. To address this, we propose a test-time self-supervised depth refinement framework that operates without ground-truth depth labels. First, we leverage diffusion models for relighting and input augmentation, incorporating Shape-from-Shading priors. Second, we integrate joint geometric-photometric modeling into Score Distillation Sampling (SDS). Third, we adopt a targeted optimization strategy: freezing the backbone encoder while fine-tuning only the decoder and optimizing intermediate implicit embeddings—thereby preventing optimization collapse. Evaluated across multiple benchmarks, our method substantially improves depth accuracy and geometric consistency, significantly outperforming DA-V2. This demonstrates the efficacy of generative self-supervision in enhancing geometric reasoning for monocular depth estimation.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Monocular depth estimation remains challenging as recent foundation models, such as Depth Anything V2 (DA-V2), struggle with real-world images that are far from the training distribution. We introduce Re-Depth Anything, a test-time self-supervision framework that bridges this domain gap by fusing DA-V2 with the powerful priors of large-scale 2D diffusion models. Our method performs label-free refinement directly on the input image by re-lighting predicted depth maps and augmenting the input. This re-synthesis method replaces classical photometric reconstruction by leveraging shape from shading (SfS) cues in a new, generative context with Score Distillation Sampling (SDS). To prevent optimization collapse, our framework employs a targeted optimization strategy: rather than optimizing depth directly or fine-tuning the full model, we freeze the encoder and only update intermediate embeddings while also fine-tuning the decoder. Across diverse benchmarks, Re-Depth Anything yields substantial gains in depth accuracy and realism over the DA-V2, showcasing new avenues for self-supervision by augmenting geometric reasoning.

Problem

• Refines monocular depth estimation for real-world images
• Bridges domain gaps using self-supervised test-time refinement
• Enhances depth accuracy via generative re-lighting and optimization

Innovation

• Self-supervised test-time refinement via re-lighting
• Fuses depth model with diffusion priors using SDS
• Optimizes only embeddings and decoder, freezes encoder