SceneTransporter: Optimal Transport-Guided Compositional Latent Diffusion for Single-Image Structured 3D Scene Generation

2026-02-26

AI Summary

Existing methods for generating structured 3D scenes from a single image struggle to organize part-level 3D objects into distinct, coherent instances in open-world settings, often resulting in part entanglement and object fragmentation. This work proposes a novel approach that formulates scene generation as a global correspondence assignment problem. By integrating an entropy-regularized optimal transport objective into the denoising process of a compositional Diffusion Transformer (DiT), the method enforces one-to-one exclusive routing between image patches and part-level 3D latent variables. Furthermore, edge-aware cost regularization and debiased clustering probes encourage semantically similar patches to coalesce into unified objects. As the first effort to incorporate optimal transport into compositional diffusion models, the proposed exclusive attention gating and competitive grouping mechanism significantly enhances instance consistency and geometric fidelity, outperforming current state-of-the-art methods in open-world 3D scene generation.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Learning to Search
• Natural Language Processing: Generation

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

We introduce SceneTransporter, an end-to-end framework for structured 3D scene generation from a single image. While existing methods generate part-level 3D objects, they often fail to organize these parts into distinct instances in open-world scenes. Through a debiased clustering probe, we reveal a critical insight: this failure stems from the lack of structural constraints within the model's internal assignment mechanism. Based on this finding, we reframe the task of structured 3D scene generation as a global correlation assignment problem. To solve this, SceneTransporter formulates and solves an entropic Optimal Transport (OT) objective within the denoising loop of the compositional DiT model. This formulation imposes two powerful structural constraints. First, the resulting transport plan gates cross-attention to enforce an exclusive, one-to-one routing of image patches to part-level 3D latents, preventing entanglement. Second, the competitive nature of the transport encourages the grouping of similar patches, a process that is further regularized by an edge-based cost, to form coherent objects and prevent fragmentation. Extensive experiments show that SceneTransporter outperforms existing methods on open-world scene generation, significantly improving instance-level coherence and geometric fidelity. Code and models will be publicly available at https://2019epwl.github.io/SceneTransporter/.

Problem

• structured 3D scene generation
• single-image 3D reconstruction
• instance-level coherence
• part-level 3D objects
• open-world scenes

Innovation

• Optimal Transport
• Latent Diffusion
• Structured 3D Scene Generation
• Compositional DiT
• Instance Coherence