DepthAgent: Towards Better Universal Depth Estimation via Sample-wise Expert Selection

2026-05-22

AI Summary

Monocular depth estimation exhibits limited generalization when deployed across diverse camera types, such as perspective, fisheye, and panoramic. To address this, this work proposes a vision-language agent that dynamically selects or fuses multiple frozen depth-expert models at the sample level by analyzing scene semantics and camera-specific characteristics. The approach employs multi-reward reinforcement learning to fine-tune a discrete decision process, uncovering complementary strengths among experts across different input domains. Experimental results demonstrate that the proposed strategy consistently outperforms single-model baselines, fixed fusion schemes, and alternative selection mechanisms across multiple camera-image benchmarks, with particularly significant gains on challenging samples.

Abstract

Monocular metric depth estimation has achieved strong progress with large-scale training and universal-camera modeling, yet robust deployment across diverse camera settings, such as perspective, fisheye, and panoramic images, remains challenging. Existing methods typically rely on a single depth estimator, overlooking that different models encode different camera assumptions and perform best under different input domains. In this paper, we show that depth experts exhibit strong sample-wise complementarity: model preference is highly correlated with camera geometry, and multi-model fusion brings the largest gains on difficult samples where individual experts are unreliable. Motivated by these observations, we propose \textbf{\ours}, a vision-language agent for adaptive monocular depth estimation. DepthAgent treats existing depth models as frozen tools and learns to analyze scene and camera cues, invoke suitable experts through multi-turn tool utilization, and select or fuse their predictions for each input. To optimize such discrete decision-making toward dense geometric quality, we design a multi-reward reinforcement fine-tuning scheme that jointly encourages valid tool execution, camera/scene analysis, expert-selection quality, and inference efficiency. Extensive experiments across perspective, fisheye, and panoramic benchmarks show that \ours consistently outperforms individual experts, fixed model fusion, and different selection strategies, with strong improvements on challenging samples, highlighting the critical role of expert selection and fusion. The code and model will be released upon publication.

Problem

• monocular depth estimation
• universal depth estimation
• camera geometry
• expert selection
• cross-domain robustness

Innovation

• depth estimation
• expert selection
• vision-language agent
• multi-model fusion
• reinforcement fine-tuning