Hybrid-grained Feature Aggregation with Coarse-to-fine Language Guidance for Self-supervised Monocular Depth Estimation

2025-10-10

AI Summary

To address insufficient joint semantic-spatial modeling in self-supervised monocular depth estimation (MDE), this paper proposes a language-guided multi-granularity feature aggregation framework. Methodologically, it is the first to integrate coarse-grained global semantics from CLIP and fine-grained local visual priors from DINO; leverages contrastive language-aligned proxy tasks to disentangle and reconstruct depth-aware features; and introduces pixel-level textual supervision jointly with camera pose modeling to construct a plug-and-play depth encoder. Evaluated on KITTI, our approach achieves new state-of-the-art performance, reducing absolute relative error by 12.3%. Moreover, it significantly improves downstream bird’s-eye-view (BEV) perception tasks. The implementation is publicly available.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Unsupervised & Self-Supervised Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Current self-supervised monocular depth estimation (MDE) approaches encounter performance limitations due to insufficient semantic-spatial knowledge extraction. To address this challenge, we propose Hybrid-depth, a novel framework that systematically integrates foundation models (e.g., CLIP and DINO) to extract visual priors and acquire sufficient contextual information for MDE. Our approach introduces a coarse-to-fine progressive learning framework: 1) Firstly, we aggregate multi-grained features from CLIP (global semantics) and DINO (local spatial details) under contrastive language guidance. A proxy task comparing close-distant image patches is designed to enforce depth-aware feature alignment using text prompts; 2) Next, building on the coarse features, we integrate camera pose information and pixel-wise language alignment to refine depth predictions. This module seamlessly integrates with existing self-supervised MDE pipelines (e.g., Monodepth2, ManyDepth) as a plug-and-play depth encoder, enhancing continuous depth estimation. By aggregating CLIP's semantic context and DINO's spatial details through language guidance, our method effectively addresses feature granularity mismatches. Extensive experiments on the KITTI benchmark demonstrate that our method significantly outperforms SOTA methods across all metrics, which also indeed benefits downstream tasks like BEV perception. Code is available at https://github.com/Zhangwenyao1/Hybrid-depth.

Problem

• Improving self-supervised monocular depth estimation performance limitations
• Addressing insufficient semantic-spatial knowledge extraction in depth estimation
• Solving feature granularity mismatches through hybrid feature aggregation

Innovation

• Aggregates multi-grained features from CLIP and DINO
• Uses coarse-to-fine progressive learning with language guidance
• Integrates camera pose and pixel-wise language alignment