DreamCube: 3D Panorama Generation via Multi-plane Synchronization

2025-06-20

AI Summary

To address geometric incompatibility and multi-view inconsistency arising from reliance on 2D foundation models in 3D panoramic synthesis—due to scarce 3D panoramic data—this paper proposes a diffusion-based multi-plane synchronous RGB-D joint generation framework. Methodologically, it introduces (1) a novel spherical coordinate alignment module coupled with a cross-plane feature synchronization operator, enabling seamless transfer of 2D visual priors into the spherical panoramic domain; and (2) an RGB-D joint diffusion modeling mechanism that simultaneously optimizes texture photorealism and depth-geometry consistency. Evaluated on panoramic image generation, depth estimation, and 3D reconstruction, the method achieves state-of-the-art performance across all tasks. It significantly improves omnidirectional content in terms of appearance diversity, geometric accuracy, and view consistency.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

3D panorama synthesis is a promising yet challenging task that demands high-quality and diverse visual appearance and geometry of the generated omnidirectional content. Existing methods leverage rich image priors from pre-trained 2D foundation models to circumvent the scarcity of 3D panoramic data, but the incompatibility between 3D panoramas and 2D single views limits their effectiveness. In this work, we demonstrate that by applying multi-plane synchronization to the operators from 2D foundation models, their capabilities can be seamlessly extended to the omnidirectional domain. Based on this design, we further introduce DreamCube, a multi-plane RGB-D diffusion model for 3D panorama generation, which maximizes the reuse of 2D foundation model priors to achieve diverse appearances and accurate geometry while maintaining multi-view consistency. Extensive experiments demonstrate the effectiveness of our approach in panoramic image generation, panoramic depth estimation, and 3D scene generation.

Problem

• Extending 2D foundation models to 3D panorama generation
• Overcoming incompatibility between 3D panoramas and 2D views
• Ensuring diverse appearance and accurate geometry in 3D synthesis

Innovation

• Multi-plane synchronization extends 2D to omnidirectional
• DreamCube uses RGB-D diffusion for 3D panoramas
• Reuses 2D priors for appearance and geometry accuracy