Unified Panoramic Geometry Estimation via Multi-View Foundation Models

2026-05-25

AI Summary

This work addresses the challenge of efficiently reconstructing complete 360° 3D scene geometry from a single panoramic image. To this end, it introduces PaGeR, a unified framework that, for the first time, enables a single model to jointly process both perspective and panoramic images. Within a single forward pass, PaGeR simultaneously predicts scale-agnostic depth, metric depth, surface normals, and a sky mask, requiring only minimal architectural modifications. By integrating a pretrained Transformer-based backbone with a hybrid training strategy, the method effectively preserves strong 3D priors. Experimental results demonstrate that PaGeR achieves state-of-the-art performance across diverse indoor and outdoor scenes and exhibits exceptional zero-shot generalization capabilities.

Abstract

Geometry estimation from perspective images has greatly advanced, maturing to the point where off-the-shelf foundation models are able to reconstruct 3D scene structure not only from multi-view imagery, but even from a single view. A natural extension is 3D reconstruction from panoramas, with the exciting prospect of recovering a full 360-degree scene from a single panoramic image. In this work, we introduce PaGeR (Panoramic Geometry Reconstruction), a framework to lift powerful 3D foundation models designed for perspective imagery to the panorama domain. Our strategy is to start from a pre-trained transformer for 3D reconstruction and turn it into a unified high-performance model that predicts scale-invariant depth, metric depth, surface normals, and sky masks from both perspective and omnidirectional images, in a single forward pass. By keeping architectural changes to a minimum and mixing perspective and panoramic images during training, PaGeR retains the rich 3D prior of the underlying foundation model while learning to also estimate geometrically consistent 360-degree scenes from single panoramas. We extensively test our method in both indoor and outdoor environments and find that it delivers state-of-the-art performance and excellent zero-shot performance across a wide range of scenes.

Problem

• panoramic geometry estimation
• 3D reconstruction
• foundation models
• omnidirectional images
• depth estimation

Innovation

• panoramic geometry estimation
• 3D foundation models
• unified reconstruction framework
• zero-shot generalization
• multi-view learning