AssemLM: Spatial Reasoning Multimodal Large Language Models for Robotic Assembly

2026-04-10

AI Summary

Current vision-language models lack precise spatial reasoning capabilities over 3D geometry, limiting their effectiveness in robotic fine assembly tasks. This work proposes a multimodal large language model that integrates assembly manuals, point clouds, and textual instructions, employing a dedicated point cloud encoder to extract geometric and rotational features. It introduces fine-grained 3D spatial reasoning into this framework for the first time and presents AssemBench, the first large-scale 6D pose benchmark specifically designed for assembly tasks. The method achieves state-of-the-art performance in 6D pose estimation across diverse assembly scenarios and successfully executes multi-step fine assembly on a real robot, significantly enhancing the model’s explicit geometric understanding of the assembly process.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Spatial reasoning is a fundamental capability for embodied intelligence, especially for fine-grained manipulation tasks such as robotic assembly. While recent vision-language models (VLMs) exhibit preliminary spatial awareness, they largely rely on coarse 2D perception and lack the ability to perform accurate reasoning over 3D geometry, which is crucial for precise assembly operations. To address this limitation, we propose AssemLM, a spatial multimodal large language model tailored for robotic assembly. AssemLM integrates assembly manuals, point clouds, and textual instructions to reason about and predict task-critical 6D assembly poses, enabling explicit geometric understanding throughout the assembly process. To effectively bridge raw 3D perception and high-level reasoning, we adopt a specialized point cloud encoder to capture fine-grained geometric and rotational features, which are then integrated into the multimodal language model to support accurate 3D spatial reasoning for assembly tasks. In addition, we construct AssemBench, a large-scale dataset and benchmark for assembly-oriented spatial reasoning, comprising over 900K multimodal samples with precise 6D pose annotations. AssemBench extends spatial reasoning evaluation beyond 2D and grounding tasks into full 3D geometric inference, filling a critical gap in existing embodied AI benchmarks. Extensive experiments demonstrate that AssemLM achieves state-of-the-art performance in 6D pose reasoning across diverse assembly scenarios. Furthermore, real-robot evaluations show that our model can support fine-grained and multi-step assembly execution in real-world settings, demonstrating its potential for robotic assembly applications.

Problem

• spatial reasoning
• robotic assembly
• 3D geometry
• 6D pose
• multimodal large language models

Innovation

• spatial reasoning
• multimodal large language model
• 6D pose estimation
• point cloud encoding
• robotic assembly