Lightweight Prompt-Guided CLIP Adaptation for Monocular Depth Estimation

2026-04-01

AI Summary

This work addresses the challenge of efficiently leveraging semantic features from pretrained vision-language models, such as CLIP, for monocular depth estimation. The authors propose MoA-DepthCLIP, a framework that employs a lightweight Mixture-of-Adapters architecture coupled with a global semantic prompt-guided mechanism to achieve spatially aware CLIP adaptation while fine-tuning only a minimal number of parameters. The method further integrates a hybrid prediction head combining depth interval classification and direct regression, along with a composite geometric constraint loss, to significantly enhance geometric accuracy. Evaluated on NYU Depth V2, the approach improves the δ₁ accuracy from 0.390 to 0.745 and reduces RMSE from 1.176 to 0.520, achieving state-of-the-art performance with remarkably low training overhead.

Abstract

Leveraging the rich semantic features of vision-language models (VLMs) like CLIP for monocular depth estimation tasks is a promising direction, yet often requires extensive fine-tuning or lacks geometric precision. We present a parameter-efficient framework, named MoA-DepthCLIP, that adapts pretrained CLIP representations for monocular depth estimation with minimal supervision. Our method integrates a lightweight Mixture-of-Adapters (MoA) module into the pretrained Vision Transformer (ViT-B/32) backbone combined with selective fine-tuning of the final layers. This design enables spatially-aware adaptation, guided by a global semantic context vector and a hybrid prediction architecture that synergizes depth bin classification with direct regression. To enhance structural accuracy, we employ a composite loss function that enforces geometric constraints. On the NYU Depth V2 benchmark, MoA-DepthCLIP achieves competitive results, significantly outperforming the DepthCLIP baseline by improving the $δ_1$ accuracy from 0.390 to 0.745 and reducing the RMSE from 1.176 to 0.520. These results are achieved while requiring substantially few trainable parameters, demonstrating that lightweight, prompt-guided MoA is a highly effective strategy for transferring VLM knowledge to fine-grained monocular depth estimation tasks.

Problem

• monocular depth estimation
• vision-language models
• CLIP adaptation
• geometric precision
• parameter efficiency

Innovation

• Mixture-of-Adapters
• prompt-guided adaptation
• monocular depth estimation
• vision-language models
• parameter-efficient tuning