SGFormer: Spherical Geometry Transformer for 360 Depth Estimation

2024-04-23

arXiv.org

AI Summary

To address severe polar distortion, structural degradation, and the difficulty of jointly modeling global and local geometry in 360° panoramic depth estimation, this paper proposes a Vision Transformer architecture incorporating spherical geometric priors. Our method introduces two key innovations: (1) the Spherical Prior Decoder (SPDecoder), the first of its kind, which explicitly encodes spherical isometry, continuity, and curvature awareness via bipolar reprojection, equivariant ring-based rotation embeddings, and geodesic-aware local distance encoding; and (2) a query-driven multi-scale conditional positional encoding scheme to enhance geometric awareness. Evaluated on multiple mainstream 360° depth datasets, our approach achieves significant improvements over state-of-the-art methods—particularly reducing polar-region depth error by a large margin—while markedly enhancing structural integrity and boundary sharpness of predicted depth maps.

Technology Category

• Computer Vision: 3D Computer Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

Panoramic distortion poses a significant challenge in 360 depth estimation, particularly pronounced at the north and south poles. Existing methods either adopt a bi-projection fusion strategy to remove distortions or model long-range dependencies to capture global structures, which can result in either unclear structure or insufficient local perception. In this paper, we propose a spherical geometry transformer, named SGFormer, to address the above issues, with an innovative step to integrate spherical geometric priors into vision transformers. To this end, we retarget the transformer decoder to a spherical prior decoder (termed SPDecoder), which endeavors to uphold the integrity of spherical structures during decoding. Concretely, we leverage bipolar re-projection, circular rotation, and curve local embedding to preserve the spherical characteristics of equidistortion, continuity, and surface distance, respectively. Furthermore, we present a query-based global conditional position embedding to compensate for spatial structure at varying resolutions. It not only boosts the global perception of spatial position but also sharpens the depth structure across different patches. Finally, we conduct extensive experiments on popular benchmarks, demonstrating our superiority over state-of-the-art solutions.

Problem

• Addresses panoramic distortion in 360 depth estimation
• Integrates spherical geometric priors into vision transformers
• Enhances global perception and local depth structure accuracy

Innovation

• Spherical geometry transformer integration
• Bipolar re-projection technique
• Query-based global conditional embedding