IDAGC: Adaptive Generalized Human-Robot Collaboration via Human Intent Estimation and Multimodal Policy Learning

2025-07-06

AI Summary

To address the core challenges in human–robot collaboration (HRC)—inaccurate intent recognition, rigid collaboration modes, and difficulty in dynamic mode switching—this paper proposes the IDAGC framework. It integrates multimodal inputs—including vision, language, force sensing, and robot state—into a unified architecture comprising multiple encoders, a Transformer-based decoder, and a conditional variational autoencoder (CVAE) for intent inference. Notably, this work is the first to jointly leverage CVAE and Transformer architectures for multimodal intent recognition and autonomous collaboration-mode reasoning, enabling cross-task policy learning and online optimization of compliant control. Experimental results demonstrate significant improvements: a +12.7% gain in intent recognition accuracy and enhanced collaboration fluency. The framework exhibits strong generalization and adaptability across diverse HRC scenarios, including assembly and object transportation.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Multi-modal Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

In Human-Robot Collaboration (HRC), which encompasses physical interaction and remote cooperation, accurate estimation of human intentions and seamless switching of collaboration modes to adjust robot behavior remain paramount challenges. To address these issues, we propose an Intent-Driven Adaptive Generalized Collaboration (IDAGC) framework that leverages multimodal data and human intent estimation to facilitate adaptive policy learning across multi-tasks in diverse scenarios, thereby facilitating autonomous inference of collaboration modes and dynamic adjustment of robotic actions. This framework overcomes the limitations of existing HRC methods, which are typically restricted to a single collaboration mode and lack the capacity to identify and transition between diverse states. Central to our framework is a predictive model that captures the interdependencies among vision, language, force, and robot state data to accurately recognize human intentions with a Conditional Variational Autoencoder (CVAE) and automatically switch collaboration modes. By employing dedicated encoders for each modality and integrating extracted features through a Transformer decoder, the framework efficiently learns multi-task policies, while force data optimizes compliance control and intent estimation accuracy during physical interactions. Experiments highlights our framework's practical potential to advance the comprehensive development of HRC.

Problem

• Accurate human intent estimation in HRC
• Seamless switching of collaboration modes
• Adaptive policy learning across multi-tasks

Innovation

• Multimodal data fusion for intent estimation
• CVAE-based human intention recognition
• Transformer decoder for policy integration