Pair2Scene: Learning Local Object Relations for Procedural Scene Generation

2026-04-13

AI Summary

This work addresses the challenge of high-fidelity 3D indoor scene generation under data scarcity and difficulties in modeling complex spatial relationships, particularly the performance degradation when generalizing to dense or out-of-distribution scenes. To this end, the authors propose a procedural generation framework that integrates learned local regularities, hierarchical scene structure, and physical constraints. The approach models object layouts through local dependencies, incorporating a pairwise object relationship learning and reasoning mechanism, along with a conditional spatial distribution estimation network, hierarchical recursive construction, collision-aware rejection sampling, and joint semantic–physical constraints. Experiments on the 3D-Pairs dataset demonstrate that the method significantly outperforms existing approaches in generating complex out-of-distribution scenes while maintaining strong physical plausibility and semantic consistency.

Technology Category

• Natural Language Processing: Generation
• Humans and AI: Game Design — Procedural Content Generation & Storytelling
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Generating high-fidelity 3D indoor scenes remains a significant challenge due to data scarcity and the complexity of modeling intricate spatial relations. Current methods often struggle to scale beyond training distribution to dense scenes or rely on LLMs/VLMs that lack the ability for precise spatial reasoning. Building on top of the observation that object placement relies mainly on local dependencies instead of information-redundant global distributions, in this paper, we propose Pair2Scene, a novel procedural generation framework that integrates learned local rules with scene hierarchies and physics-based algorithms. These rules mainly capture two types of inter-object relations, namely support relations that follow physical hierarchies, and functional relations that reflect semantic links. We model these rules through a network, which estimates spatial position distributions of dependent objects conditioned on position and geometry of the anchor ones. Accordingly, we curate a dataset 3D-Pairs from existing scene data to train the model. During inference, our framework can generate scenes by recursively applying our model within a hierarchical structure, leveraging collision-aware rejection sampling to align local rules into coherent global layouts. Extensive experiments demonstrate that our framework outperforms existing methods in generating complex environments that go beyond training data while maintaining physical and semantic plausibility.

Problem

• 3D scene generation
• spatial relations
• data scarcity
• physical plausibility
• semantic plausibility

Innovation

• procedural scene generation
• local object relations
• support relations
• functional relations
• hierarchical scene synthesis