CL-MVSNet: Unsupervised Multi-view Stereo with Dual-level Contrastive Learning

2023-10-01

IEEE International Conference on Computer Vision

AI Summary

To address incomplete and brittle depth estimation in unsupervised multi-view stereo (MVS) caused by low-texture regions and view-dependent effects (e.g., reflections), this paper proposes a two-level contrastive learning framework: image-level and scene-level contrastive branches are jointly optimized to enhance contextual awareness and feature representation robustness. Additionally, we introduce an L₀.₅ photometric consistency loss that selectively emphasizes high-confidence correspondences, mitigating the over-penalization of low-gradient regions inherent in conventional L₁/L₂ losses. The method is fully unsupervised—requiring no ground-truth depth annotations. Evaluated on DTU and Tanks & Temples benchmarks, it achieves state-of-the-art performance among unsupervised MVS approaches and surpasses leading supervised methods without fine-tuning. Our core contributions are the first-ever dual-granularity contrastive mechanism for MVS and an L₀.₅ norm-driven photometric constraint, jointly advancing robustness and accuracy in texture-deficient and view-dependent scenarios.

Technology Category

• Computer Vision: Learning & Optimization for CV
• Machine Learning: Multi-instance/Multi-view Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Unsupervised Multi-View Stereo (MVS) methods have achieved promising progress recently. However, previous methods primarily depend on the photometric consistency assumption, which may suffer from two limitations: indistinguishable regions and view-dependent effects, e.g., low-textured areas and reflections. To address these issues, in this paper, we propose a new dual-level contrastive learning approach, named CL-MVSNet. Specifically, our model integrates two contrastive branches into an unsupervised MVS framework to construct additional supervisory signals. On the one hand, we present an image-level contrastive branch to guide the model to acquire more context awareness, thus leading to more complete depth estimation in indistinguishable regions. On the other hand, we exploit a scene-level contrastive branch to boost the representation ability, improving robustness to view-dependent effects. Moreover, to recover more accurate 3D geometry, we introduce an ℒ0.5 photometric consistency loss, which encourages the model to focus more on accurate points while mitigating the gradient penalty of undesirable ones. Extensive experiments on DTU and Tanks&Temples benchmarks demonstrate that our approach achieves state-of-the-art performance among all end-to-end unsupervised MVS frameworks and outperforms its supervised counterpart by a considerable margin without fine-tuning.

Problem

• Addresses limitations in photometric consistency in MVS
• Improves depth estimation in low-textured and reflective areas
• Enhances robustness to view-dependent effects in 3D reconstruction

Innovation

• Dual-level contrastive learning for MVS
• Image-level and scene-level contrastive branches
• L0.5 photometric consistency loss for accuracy