PixWorld: Unifying 3D Scene Generation and Reconstruction in Pixel Space

2026-07-06

AI Summary

Existing methods for 3D scene generation and reconstruction rely on disparate paradigms, often requiring pretrained encoders, suffering from information loss, and lacking a unified optimization objective. This work proposes PixWorld, the first unified diffusion framework operating directly in pixel space, which jointly addresses generation and reconstruction by applying diffusion supervision to rendered images. By eschewing latent-space encoding, PixWorld integrates differentiable rendering with geometry-aware losses derived from pretrained 3D foundation models to enhance structural consistency. Experiments demonstrate that PixWorld outperforms existing latent-space approaches in 3D generation and achieves state-of-the-art performance in reconstruction, validating the efficacy of a unified pixel-space paradigm.

Abstract

3D reconstruction and generation are commonly tackled by separate paradigms: pixel-based regression for reconstruction, and latent diffusion for generation. Recent works attempt to unify them in latent space, but with notable drawbacks: the diffusion objective is defined on latent features rather than the underlying 3D representation, and both branches suffer from information loss introduced by latent encoding, while requiring a pretrained Variational Autoencoder (VAE) or Representation Autoencoder (RAE). In this paper, we reformulate these two tasks under a unified pixel-space diffusion paradigm and introduce PixWorld, a single model that jointly addresses 3D reconstruction and generation. By supervising diffusion directly on rendered images, PixWorld removes the above limitations and aligns optimization with 3D scene fidelity. Beyond photometric and perceptual supervision that operates at the 2D image level and lacks 3D geometric awareness, we further introduce a geometry perception loss that aligns rendered views with their ground truth in the geometry-aware feature space of a pretrained 3D foundation model, providing 3D structural supervision. PixWorld consistently outperforms prior latent-space generation methods and matches state-of-the-art reconstruction methods, demonstrating the superiority of a unified pixel-space approach.

Problem

• 3D reconstruction
• 3D generation
• latent space
• pixel-space diffusion
• unified paradigm

Innovation

• pixel-space diffusion
• unified 3D generation and reconstruction
• geometry perception loss
• rendered image supervision
• 3D foundation model