BrickNet: Graph-Backed Generative Brick Assembly

2026-04-24

AI Summary

Existing approaches struggle to generate complex LEGO models comprising thousands of semantically diverse parts, being largely confined to simple, discrete voxel-like structures. To address this limitation, this work introduces a large-scale dataset of human-designed LDraw brick assemblies and proposes a graph-based procedural representation that explicitly models spatial connectivity among parts. This representation guides a language model to generate physically plausible assembly sequences while respecting geometric and mechanical constraints. Rather than directly predicting 3D poses, the method parameterizes part connectivity through graph structures, significantly enhancing both the physical feasibility and structural diversity of generated models. Experiments demonstrate that the system efficiently produces complex, valid LEGO constructions supporting a wide variety of brick types. The dataset and model are publicly released to foster further research in procedural generation and physical reasoning.

Technology Category

• Humans and AI: Game Design — Procedural Content Generation & Storytelling
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Natural Language Processing: Generation

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

We train a language model to generate LEGO-brick build sequences. While prior work has been restricted to discrete, voxel-like towers, we consider a much broader set of pieces, encompassing thousands of part types with diverse connection semantics. To enable this, we first collect a large-scale dataset of over 100,000 human-designed LDraw brick objects and scenes. The complexity of our setting makes it challenging to autoregressively assemble structures that satisfy physical constraints. When predicting block pose directly, build sequences quickly become invalid after a small number of steps. Although pieces are placed in 3D space, it is the spatial relationships of the parts which define the whole. With this in mind, we design a graph-based program representation that parametrizes structure through connectivity, improving the physical grounding of generated sequences. To enable future applications, we make our dataset and models available for research purposes. https://kulits.github.io/BrickNet

Problem

• LEGO assembly
• generative modeling
• physical constraints
• brick connectivity
• 3D structure generation

Innovation

• graph-based representation
• generative brick assembly
• physical grounding
• LDraw dataset
• connectivity-aware modeling