SphereDiff: Tuning-free Omnidirectional Panoramic Image and Video Generation via Spherical Latent Representation

2025-04-19

AI Summary

To address severe polar distortion and boundary discontinuity in 360° omnidirectional image/video generation under equirectangular projection (ERP), this paper proposes the first fine-tuning-free spherical latent-space diffusion framework. Methodologically, we design a spherical latent variable representation, develop a spherical MultiDiffusion sampling mechanism, and introduce an ERP-distortion-aware weighted fusion strategy to achieve globally uniform and seamless latent-space modeling and reconstruction. Unlike prior approaches relying on ERP grids or post-processing, our method inherently avoids projection-induced distortions at the representation level, significantly improving polar coherence and global geometric fidelity without any fine-tuning. Experiments demonstrate state-of-the-art performance across panoramic-specific metrics—including PSNR, SSIM, and Sphere-FID—while exhibiting strong practicality and generalization for high-resolution AR/VR content generation.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Learning with Manifolds
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Responsible Web: Data and user privacy-enhancing technologies for the Web
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

The increasing demand for AR/VR applications has highlighted the need for high-quality 360-degree panoramic content. However, generating high-quality 360-degree panoramic images and videos remains a challenging task due to the severe distortions introduced by equirectangular projection (ERP). Existing approaches either fine-tune pretrained diffusion models on limited ERP datasets or attempt tuning-free methods that still rely on ERP latent representations, leading to discontinuities near the poles. In this paper, we introduce SphereDiff, a novel approach for seamless 360-degree panoramic image and video generation using state-of-the-art diffusion models without additional tuning. We define a spherical latent representation that ensures uniform distribution across all perspectives, mitigating the distortions inherent in ERP. We extend MultiDiffusion to spherical latent space and propose a spherical latent sampling method to enable direct use of pretrained diffusion models. Moreover, we introduce distortion-aware weighted averaging to further improve the generation quality in the projection process. Our method outperforms existing approaches in generating 360-degree panoramic content while maintaining high fidelity, making it a robust solution for immersive AR/VR applications. The code is available here. https://github.com/pmh9960/SphereDiff

Problem

• Generating high-quality 360-degree panoramic content without distortions
• Overcoming discontinuities near poles in equirectangular projection methods
• Enabling seamless panoramic image and video generation for AR/VR

Innovation

• Spherical latent representation for uniform distribution
• MultiDiffusion extended to spherical latent space
• Distortion-aware weighted averaging improves quality