Unified Panoramic-Gaussian Representation for Monocular 4D Scene Synthesis

2026-07-01

AI Summary

Existing methods for 4D scene synthesis from monocular videos are constrained by view interpolation, often failing to ensure cross-view consistency, particularly in out-of-view regions. This work proposes PanoGaussian, a unified framework that integrates panoramic trajectory guidance with an explicit dynamic Gaussian representation. By incorporating 3D dynamic physical priors into the modeling process, the method effectively mitigates scale and deformation artifacts in unseen regions. PanoGaussian enables camera-conditioned video generation, producing high-quality, temporally coherent 4D dynamic scenes even under large viewpoint changes, significantly outperforming current state-of-the-art approaches.

Abstract

4D scene synthesis from monocular videos has made significant progress in recent years. However, existing methods are typically constrained by view interpolation. As a result, they struggle to infer unseen regions beyond the observed views. In this paper, we reformulate the task as 4D scene synthesis with unseen regions, which extends beyond traditional interpolation settings. Camera-conditioned video generation enables unseen region synthesis by guiding generation along specified cameras. However, these methods lack explicit 3D priors and are optimized with random camera trajectories. This design leads to severe inconsistencies under large trajectory deviations. To address this limitation, we build a unified training and inference framework with panoramic trajectory guidance. While this design improves cross-view consistency, the panoramic representation alone fails to model dynamic content effectively. Object motion in panoramic space introduces scale and shape distortions. To address this, we propose PanoGaussian, a unified Panoramic-Gaussian representation that distills the panoramic representation into an explicit dynamic Gaussian representation to capture dynamic physical priors of the 4D scene. Experiments demonstrate that PanoGaussian achieves consistent 4D scene synthesis even under large viewpoint variations.

Problem

• 4D scene synthesis
• unseen region synthesis
• cross-view consistency
• dynamic content modeling
• monocular video

Innovation

• PanoGaussian
• 4D scene synthesis
• panoramic representation
• dynamic Gaussian representation
• unseen region synthesis