MIDI: Multi-Instance Diffusion for Single Image to 3D Scene Generation

2024-12-04

arXiv.org

AI Summary

This work addresses the end-to-end generation of multi-instance 3D scenes from a single input image, with emphasis on spatial relationship accuracy and cross-scene generalization. We propose the first diffusion-based framework explicitly designed for multi-instance 3D scene modeling, introducing a novel multi-instance attention mechanism that jointly captures inter-object geometric interactions and global spatial consistency during the denoising process—eliminating the need for sequential generation or post-hoc layout optimization. Built upon a pre-trained image-to-3D object model, our framework integrates local object-centric representations with global scene context, while jointly applying scene-level supervision and per-object data regularization. Evaluated on synthetic data, real-world images, and text-to-image–style stylized inputs, our method achieves state-of-the-art performance, significantly improving 3D scene completeness, spatial fidelity, and generalization capability across diverse scenarios.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Natural Language Processing: Generation

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

This paper introduces MIDI, a novel paradigm for compositional 3D scene generation from a single image. Unlike existing methods that rely on reconstruction or retrieval techniques or recent approaches that employ multi-stage object-by-object generation, MIDI extends pre-trained image-to-3D object generation models to multi-instance diffusion models, enabling the simultaneous generation of multiple 3D instances with accurate spatial relationships and high generalizability. At its core, MIDI incorporates a novel multi-instance attention mechanism, that effectively captures inter-object interactions and spatial coherence directly within the generation process, without the need for complex multi-step processes. The method utilizes partial object images and global scene context as inputs, directly modeling object completion during 3D generation. During training, we effectively supervise the interactions between 3D instances using a limited amount of scene-level data, while incorporating single-object data for regularization, thereby maintaining the pre-trained generalization ability. MIDI demonstrates state-of-the-art performance in image-to-scene generation, validated through evaluations on synthetic data, real-world scene data, and stylized scene images generated by text-to-image diffusion models.

Problem

• Generating 3D scenes from single images with accurate spatial relationships
• Extending single-object 3D models to multi-instance diffusion for scene generation
• Capturing inter-object interactions without complex multi-step processes

Innovation

• Extends image-to-3D models to multi-instance diffusion
• Uses multi-instance attention for inter-object interactions
• Combines scene-level and single-object data for training