Prompt-to-Product: Generative Assembly via Bimanual Manipulation

2025-08-28

AI Summary

Automating physical assembly from natural-language instructions remains a challenging open problem in robotics and AI. Method: This paper introduces the first end-to-end text-to-physical-Lego construction system, integrating large language models (LLMs) for instruction parsing and generative 3D structural design, coupled with dual-arm robotic motion planning and coordinated control for real-world assembly. Contribution/Results: It presents the first deep integration of generative AI with a dual-robot platform to achieve a fully automated closed-loop pipeline—from natural-language prompt to tangible assembly. User studies demonstrate that the system reliably generates structurally sound designs and executes robust physical construction, significantly reducing reliance on human design expertise and domain-specific skills. By bridging high-level intent and low-level actuation, it shortens the ideation-to-implementation cycle for physical prototyping.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Agentic search

Abstract

Creating assembly products demands significant manual effort and expert knowledge in 1) designing the assembly and 2) constructing the product. This paper introduces Prompt-to-Product, an automated pipeline that generates real-world assembly products from natural language prompts. Specifically, we leverage LEGO bricks as the assembly platform and automate the process of creating brick assembly structures. Given the user design requirements, Prompt-to-Product generates physically buildable brick designs, and then leverages a bimanual robotic system to construct the real assembly products, bringing user imaginations into the real world. We conduct a comprehensive user study, and the results demonstrate that Prompt-to-Product significantly lowers the barrier and reduces manual effort in creating assembly products from imaginative ideas.

Problem

• Automating assembly product creation from natural language prompts
• Generating physically buildable designs using LEGO bricks
• Reducing manual effort with bimanual robotic construction system

Innovation

• Generates buildable LEGO designs from language prompts
• Uses bimanual robotic system for physical assembly
• Automates entire pipeline from imagination to real product