How to Evaluate Monocular Depth Estimation?

2025-10-22

AI Summary

Monocular depth estimation evaluation lacks standardization; existing metrics exhibit insufficient sensitivity to geometric distortions (e.g., curvature deformations) and poor alignment with human perception. To address this, we systematically analyze the perturbation sensitivity of prevalent evaluation metrics and validate their shortcomings via controlled human subjective experiments. We propose a novel metric based on relative surface normals—geometrically grounded and perceptually meaningful—and design an interpretable, geometry-aware depth visualization tool. Furthermore, we establish a principled multi-metric fusion framework for composite evaluation. Leveraging normal-vector geometric modeling, rigorous perturbation analysis, and perceptual validation, our metric achieves significantly improved consistency with human judgments across multiple benchmarks, yielding an average 23.6% increase in Spearman correlation coefficient. All code and datasets are publicly released, providing both theoretical foundations and practical tools toward standardized monocular depth evaluation.

Technology Category

• Computer Vision: Other Foundations of Computer Vision
• Search and Optimization: Evaluation and Analysis
• Machine Learning: Evaluation and Analysis

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Monocular depth estimation is an important task with rapid progress, but how to evaluate it remains an open question, as evidenced by a lack of standardization in existing literature and a large selection of evaluation metrics whose trade-offs and behaviors are not well understood. This paper contributes a novel, quantitative analysis of existing metrics in terms of their sensitivity to various types of perturbations of ground truth, emphasizing comparison to human judgment. Our analysis reveals that existing metrics are severely under-sensitive to curvature perturbation such as making flat surfaces wavy. To remedy this, we introduce a new metric based on relative surface normals, along with new depth visualization tools and a principled method to create composite metrics with better human alignment. Code and data are available at: https://github.com/princeton-vl/evalmde.

Problem

• Evaluating monocular depth estimation lacks standardization and metric understanding
• Existing metrics show poor sensitivity to ground truth curvature perturbations
• New metric using relative surface normals improves alignment with human judgment

Innovation

• New metric based on relative surface normals
• Developed depth visualization tools
• Created composite metrics for human alignment