CoViLLM: An Adaptive Human-Robot Collaborative Assembly Framework Using Large Language Models for Manufacturing

2026-03-11

AI Summary

This work proposes an adaptive human-robot collaborative assembly framework that overcomes the limitations of traditional manufacturing robots, which rely on predefined rules and struggle with customized or entirely new products. By integrating depth-camera-based localization, human operator recognition of novel components, and the natural language understanding capabilities of large language models (LLMs), the system dynamically generates assembly task sequences for known, customized, and previously unseen products. This approach breaks away from the fixed perception-action pipelines typical of conventional human-robot collaboration systems and, for the first time, leverages LLMs to generate real-time assembly plans for unknown products, significantly enhancing system generalization. Experiments on the NIST assembly platform demonstrate the framework’s flexibility and effectiveness in non-predefined scenarios, thereby expanding the application boundaries of human-robot collaboration in smart manufacturing.

Technology Category

• Intelligent Robots: Human-Robot Interaction
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

With increasing demand for mass customization, traditional manufacturing robots that rely on rule-based operations lack the flexibility to accommodate customized or new product variants. Human-Robot Collaboration (HRC) has demonstrated potential to improve system adaptability by leveraging human versatility and decision-making capabilities. However, existing HRC frame- works typically depend on predefined perception-manipulation pipelines, limiting their ability to autonomously generate task plans for new product assembly. In this work, we propose CoViLLM, an adaptive human-robot collaborative assembly frame- work that supports the assembly of customized and previously unseen products. CoViLLM combines depth-camera-based localization for object position estimation, human operator classification for identifying new components, and an Large Language Model (LLM) for assembly task planning based on natural language instructions. The framework is validated on the NIST Assembly Task Board for known, customized, and new product cases. Experimental results show that the proposed framework enables flexible collaborative assembly by extending HRC beyond predefined product and task settings.

Problem

• Human-Robot Collaboration
• Mass Customization
• Assembly Task Planning
• Large Language Models
• Flexible Manufacturing

Innovation

• Large Language Model
• Human-Robot Collaboration
• Adaptive Assembly
• Customized Manufacturing
• Task Planning