Robust and Flexible Omnidirectional Depth Estimation with Multiple 360-degree Cameras

2024-09-23

AI Summary

Depth estimation in multi-360° camera surround-view systems suffers from poor robustness and generalization due to lens contamination and variable camera layouts. Method: We propose a geometry-aware spherical depth estimation framework. It introduces the generalized spherical epipolar isometric rectangular projection to simplify spherical matching constraints; designs a two-stage fusion with one-stage spherical sweeping for joint depth estimation; and employs spherical feature extraction and spherical sweeping to fuse multi-view redundant information. Contribution/Results: We construct the first realistic 360° synthetic dataset featuring physically plausible lens contamination and glare (12K panoramas + 3K ground-truth depth maps). Our method achieves state-of-the-art accuracy under contaminated inputs, supports arbitrary numbers and configurations of 360° cameras, and significantly outperforms existing approaches on our dataset and multiple public benchmarks.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: 3D Computer Vision
• Machine Learning: Multi-instance/Multi-view Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Sustainability and carbon-aware systems for Web, mobile, and WoT
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Omnidirectional depth estimation has received much attention from researchers in recent years. However, challenges arise due to camera soiling and variations in camera layouts, affecting the robustness and flexibility of the algorithm. In this paper, we use the geometric constraints and redundant information of multiple 360-degree cameras to achieve robust and flexible multi-view omnidirectional depth estimation. We implement two algorithms, in which the two-stage algorithm obtains initial depth maps by pairwise stereo matching of multiple cameras and fuses the multiple depth maps to achieve the final depth estimation; the one-stage algorithm adopts spherical sweeping based on hypothetical depths to construct a uniform spherical matching cost of the multi-camera images and obtain the depth. Additionally, a generalized epipolar equirectangular projection is introduced to simplify the spherical epipolar constraints. To overcome panorama distortion, a spherical feature extractor is implemented. Furthermore, a synthetic 360-degree dataset consisting of 12K road scene panoramas and 3K ground truth depth maps is presented to train and evaluate 360-degree depth estimation algorithms. Our dataset takes soiled camera lenses and glare into consideration, which is more consistent with the real-world environment. Experiments show that our two algorithms achieve state-of-the-art performance, accurately predicting depth maps even when provided with soiled panorama inputs. The flexibility of the algorithms is experimentally validated in terms of camera layouts and numbers.

Problem

• Robust omnidirectional depth estimation
• Flexible multi-camera depth algorithms
• Overcoming camera soiling and layout variations

Innovation

• Multiple 360-degree cameras
• Geometric constraints utilization
• Spherical feature extraction