Vanishing Depth: A Depth Adapter with Positional Depth Encoding for Generalized Image Encoders

2025-03-25

AI Summary

Current vision encoders lack metric depth understanding, limiting their generalization across RGB-D downstream tasks—including semantic segmentation, depth completion, scene classification, and 6D pose estimation. To address this, we propose a fine-tuning-free self-supervised depth adaptation framework that aligns metric depth into the pre-trained RGB encoder’s feature space. We introduce a novel position-aware depth encoding mechanism that enables robust feature extraction invariant to depth density and distribution shifts. Our method achieves state-of-the-art performance across multiple benchmarks: 56.05 mIoU on SUN-RGBD semantic segmentation, 88.3 RMSE on VOID depth completion, 83.8% Top-1 accuracy on NYUv2 scene classification, and superior results on 6D pose estimation—consistently outperforming strong baselines including DINOv2, EVA-02, and Omnivore.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Generalized metric depth understanding is critical for precise vision-guided robotics, which current state-of-the-art (SOTA) vision-encoders do not support. To address this, we propose Vanishing Depth, a self-supervised training approach that extends pretrained RGB encoders to incorporate and align metric depth into their feature embeddings. Based on our novel positional depth encoding, we enable stable depth density and depth distribution invariant feature extraction. We achieve performance improvements and SOTA results across a spectrum of relevant RGBD downstream tasks - without the necessity of finetuning the encoder. Most notably, we achieve 56.05 mIoU on SUN-RGBD segmentation, 88.3 RMSE on Void's depth completion, and 83.8 Top 1 accuracy on NYUv2 scene classification. In 6D-object pose estimation, we outperform our predecessors of DinoV2, EVA-02, and Omnivore and achieve SOTA results for non-finetuned encoders in several related RGBD downstream tasks.

Problem

• Extend pretrained RGB encoders for metric depth understanding
• Enable stable depth-invariant feature extraction
• Improve performance in RGBD downstream tasks without finetuning

Innovation

• Self-supervised training for RGB encoder extension
• Positional depth encoding for stable feature extraction
• No finetuning needed for SOTA RGBD performance