Context-aware collaborative pushing of heavy objects using skeleton-based intention prediction

2025-05-15

AI Summary

In industrial human-robot collaborative pushing/pulling tasks, the absence of contact-based force sensing impedes real-time recognition of human motion intent. To address this, we propose a sensorless intent prediction method leveraging human skeletal pose sequences. Our approach innovatively integrates a context-aware mechanism with a directed graph neural network (D-GNN) to construct a spatiotemporal dynamic modeling framework, explicitly encoding directed inter-joint interactions for accurate decoding of nonverbal collaborative intent. This method eliminates reliance on physical force sensors, establishing a novel paradigm of force-sensing substitution and multimodal fusion. Experimental results demonstrate that robot assistance reduces human effort by 42%, shortens task completion time by 31%, and achieves a motion intent prediction accuracy of 91.3%.

Technology Category

• Intelligent Robots: Human-Robot Interaction
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Planning, Routing, and Scheduling: Activity and Plan Recognition

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

In physical human-robot interaction, force feedback has been the most common sensing modality to convey the human intention to the robot. It is widely used in admittance control to allow the human to direct the robot. However, it cannot be used in scenarios where direct force feedback is not available since manipulated objects are not always equipped with a force sensor. In this work, we study one such scenario: the collaborative pushing and pulling of heavy objects on frictional surfaces, a prevalent task in industrial settings. When humans do it, they communicate through verbal and non-verbal cues, where body poses, and movements often convey more than words. We propose a novel context-aware approach using Directed Graph Neural Networks to analyze spatio-temporal human posture data to predict human motion intention for non-verbal collaborative physical manipulation. Our experiments demonstrate that robot assistance significantly reduces human effort and improves task efficiency. The results indicate that incorporating posture-based context recognition, either together with or as an alternative to force sensing, enhances robot decision-making and control efficiency.

Problem

• Predict human motion intention without force feedback
• Enable collaborative pushing of heavy objects using posture data
• Improve robot decision-making with context-aware intention prediction

Innovation

• Skeleton-based intention prediction via Directed Graph Neural Networks
• Context-aware posture analysis for non-verbal collaboration
• Combining posture and force sensing for robot control