FoundationStereo: Zero-Shot Stereo Matching

2025-01-17

AI Summary

To address the weak zero-shot cross-domain generalization capability in stereo matching, this paper proposes the first foundational model paradigm for stereo vision. Methodologically, we construct a million-scale high-fidelity synthetic dataset and design a self-purifying disambiguation pipeline; introduce side-tuning to transfer monocular priors into a stereo backbone, and integrate long-range contextual modeling with learnable cost-volume filtering to significantly mitigate simulation-to-real domain shift. Without any fine-tuning on real-world data, our model achieves zero-shot state-of-the-art performance across multiple real-world benchmarks (e.g., KITTI, ETH3D), demonstrating substantially improved cross-domain robustness. This work establishes a new standard for zero-shot stereo depth estimation.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Tremendous progress has been made in deep stereo matching to excel on benchmark datasets through per-domain fine-tuning. However, achieving strong zero-shot generalization - a hallmark of foundation models in other computer vision tasks - remains challenging for stereo matching. We introduce FoundationStereo, a foundation model for stereo depth estimation designed to achieve strong zero-shot generalization. To this end, we first construct a large-scale (1M stereo pairs) synthetic training dataset featuring large diversity and high photorealism, followed by an automatic self-curation pipeline to remove ambiguous samples. We then design a number of network architecture components to enhance scalability, including a side-tuning feature backbone that adapts rich monocular priors from vision foundation models to mitigate the sim-to-real gap, and long-range context reasoning for effective cost volume filtering. Together, these components lead to strong robustness and accuracy across domains, establishing a new standard in zero-shot stereo depth estimation.

Problem

• Zero-shot Generalization
• Stereo Matching
• Depth Estimation

Innovation

• Zero-shot Generalization
• Synthetic Stereo Pairs
• Depth Estimation