PointDiT: Pixel-Space Diffusion for Monocular Geometry Estimation

2026-07-02

AI Summary

This work addresses the structural ambiguity and training complexity in single-image 3D geometry reconstruction caused by reliance on latent-space compression or hybrid architectures. We propose an extremely minimalist pixel-space diffusion Transformer that trains an end-to-end diffusion model directly on raw 3D point patches, eliminating implicit encoding, complex loss functions, and the need for a point-patch tokenizer. Leveraging pretrained DINOv2 image features as conditioning guidance for geometry generation and built upon a standard ViT backbone, our method achieves the first purely pixel-space diffusion-based geometric reconstruction. It significantly enhances geometric sharpness and robustness—particularly in transparent and highly ambiguous regions—and outperforms existing implicit diffusion approaches.

Abstract

State-of-the-art single-image 3D reconstruction methods often rely on complex hybrid architectures and loss functions, or compress geometry into latent spaces in order to leverage pre-trained latent diffusion models. In this work, we show that such architectural overhead and intricate loss formulations are unnecessary. We introduce a minimalist pixel-space Diffusion Transformer, built on a plain ViT, that operates directly on raw 3D point map patches and is conditioned on image tokens from a pre-trained DINOv3. Unlike existing latent diffusion approaches, we train our diffusion backbone entirely from scratch, eliminating the need for point map tokenizers. Despite its simplicity, our approach surpasses complex latent-based diffusion models while remaining significantly simpler than hybrid alternatives. Notably, it produces sharper geometric structure and is more robust in highly ambiguous regions, such as transparent objects.

Problem

• monocular geometry estimation
• 3D reconstruction
• diffusion models
• latent space
• single-image

Innovation

• pixel-space diffusion
• monocular geometry estimation
• Diffusion Transformer
• ViT
• DINOv3