Beyond the Frame: Generating 360{deg} Panoramic Videos from Perspective Videos

2025-04-10

AI Summary

This work addresses the challenging task of synthesizing 360° omnidirectional video from conventional perspective videos, bridging the modeling gap between narrow field-of-view inputs and ultra-wide spherical outputs while preserving high fidelity and spatiotemporal coherence. We propose a geometry- and motion-aware diffusion-based generation framework: (1) spherical convolution and reprojection alignment modules explicitly model spherical geometry; (2) motion-field-guided inter-frame consistency constraints enforce temporal coherence; and (3) a novel large-scale pipeline for curating paired perspective–360° video data is introduced. Our method generates seamless, low-distortion 360° videos from real-world footage, enabling applications including video stabilization, free-viewpoint navigation, and interactive visual question answering. Quantitative and qualitative evaluations demonstrate significant improvements in cross-view spatiotemporal consistency and perceptual quality over prior approaches.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Sustainability and carbon-aware systems for Web, mobile, and WoT
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

360{deg} videos have emerged as a promising medium to represent our dynamic visual world. Compared to the"tunnel vision"of standard cameras, their borderless field of view offers a more complete perspective of our surroundings. While existing video models excel at producing standard videos, their ability to generate full panoramic videos remains elusive. In this paper, we investigate the task of video-to-360{deg} generation: given a perspective video as input, our goal is to generate a full panoramic video that is consistent with the original video. Unlike conventional video generation tasks, the output's field of view is significantly larger, and the model is required to have a deep understanding of both the spatial layout of the scene and the dynamics of objects to maintain spatio-temporal consistency. To address these challenges, we first leverage the abundant 360{deg} videos available online and develop a high-quality data filtering pipeline to curate pairwise training data. We then carefully design a series of geometry- and motion-aware operations to facilitate the learning process and improve the quality of 360{deg} video generation. Experimental results demonstrate that our model can generate realistic and coherent 360{deg} videos from in-the-wild perspective video. In addition, we showcase its potential applications, including video stabilization, camera viewpoint control, and interactive visual question answering.

Problem

• Generating 360° panoramic videos from perspective videos
• Maintaining spatio-temporal consistency in panoramic videos
• Understanding spatial layout and object dynamics for video conversion

Innovation

• Leverages online 360° videos for training data
• Uses geometry-aware operations for scene understanding
• Incorporates motion-aware techniques for temporal consistency