ArtLLM: Generating Articulated Assets via 3D LLM

2026-03-01

AI Summary

Existing methods for generating articulated 3D assets with joint structures suffer from inefficiency, poor generalization, and geometric redundancy, hindering the creation of high-quality interactive digital environments. This work proposes ArtLLM, a novel framework that leverages a 3D multimodal large language model to autoregressively predict a variable number of parts and their associated joints directly from a complete mesh, while jointly inferring the underlying kinematic structure to guide a conditional 3D generative model toward high-fidelity geometry synthesis. By unifying the modeling of part layout and motion relationships, ArtLLM transcends the limitations of conventional optimization- or retrieval-based paradigms. Evaluated on PartNet-Mobility, the method significantly outperforms existing approaches in both part layout and joint prediction accuracy, and demonstrates successful applications in real-world object digital twins and scalable robot learning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Creating interactive digital environments for gaming, robotics, and simulation relies on articulated 3D objects whose functionality emerges from their part geometry and kinematic structure. However, existing approaches remain fundamentally limited: optimization-based reconstruction methods require slow, per-object joint fitting and typically handle only simple, single-joint objects, while retrieval-based methods assemble parts from a fixed library, leading to repetitive geometry and poor generalization. To address these challenges, we introduce ArtLLM, a novel framework for generating high-quality articulated assets directly from complete 3D meshes. At its core is a 3D multimodal large language model trained on a large-scale articulation dataset curated from both existing articulation datasets and procedurally generated objects. Unlike prior work, ArtLLM autoregressively predicts a variable number of parts and joints, inferring their kinematic structure in a unified manner from the object's point cloud. This articulation-aware layout then conditions a 3D generative model to synthesize high-fidelity part geometries. Experiments on the PartNet-Mobility dataset show that ArtLLM significantly outperforms state-of-the-art methods in both part layout accuracy and joint prediction, while generalizing robustly to real-world objects. Finally, we demonstrate its utility in constructing digital twins, highlighting its potential for scalable robot learning.

Problem

• articulated 3D objects
• 3D asset generation
• kinematic structure
• part geometry
• digital twins

Innovation

• Articulated 3D Generation
• 3D Multimodal LLM
• Kinematic Structure Prediction
• Autoregressive Part-Joint Modeling
• Digital Twin Construction