"Set It Up": Functional Object Arrangement with Compositional Generative Models

2025-08-04

AI Summary

This work addresses the challenges of ambiguous natural-language instructions and unspecified target object poses in Functional Object Rearrangement (FORM). We propose SetItUp, a neuro-symbolic framework that parses large-language-model-generated Python programs into abstract spatial relation graphs and dynamically composes pre-trained diffusion models online. This enables end-to-end mapping from few-shot demonstrations and structured linguistic descriptions to physically plausible, functionally correct, and visually harmonious object poses. Our key innovations are (i) program-guided relational modeling, which grounds symbolic spatial constraints in executable code, and (ii) a plug-and-play compositional mechanism for diffusion models, enabling flexible, task-specific generation without retraining. Evaluated on three layout benchmarks, SetItUp significantly outperforms prior methods across functional correctness, physical feasibility, and aesthetic quality—demonstrating strong generalization and practical utility in real-world FORM scenarios.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Functional object arrangement (FORM) is the task of arranging objects to fulfill a function, e.g., "set up a dining table for two". One key challenge here is that the instructions for FORM are often under-specified and do not explicitly specify the desired object goal poses. This paper presents SetItUp, a neuro-symbolic framework that learns to specify the goal poses of objects from a few training examples and a structured natural-language task specification. SetItUp uses a grounding graph, which is composed of abstract spatial relations among objects (e.g., left-of), as its intermediate representation. This decomposes the FORM problem into two stages: (i) predicting this graph among objects and (ii) predicting object poses given the grounding graph. For (i), SetItUp leverages large language models (LLMs) to induce Python programs from a task specification and a few training examples. This program can be executed to generate grounding graphs in novel scenarios. For (ii), SetItUp pre-trains a collection of diffusion models to capture primitive spatial relations and online composes these models to predict object poses based on the grounding graph. We evaluated SetItUp on a dataset spanning three distinct task families: arranging tableware on a dining table, organizing items on a bookshelf, and laying out furniture in a bedroom. Experiments show that SetItUp outperforms existing models in generating functional, physically feasible, and aesthetically pleasing object arrangements. This article extends our conference paper published at Robotics: Science and Systems (RSS) 2024.

Problem

• Functional object arrangement from under-specified instructions
• Predicting object poses using grounding graphs
• Generating feasible and aesthetic object arrangements

Innovation

• Neuro-symbolic framework for functional object arrangement
• LLM-induced Python programs generate grounding graphs
• Diffusion models compose for spatial pose prediction