Multi-Class Human/Object Detection on Robot Manipulators using Proprioceptive Sensing

2025-08-04

AI Summary

Existing contact identification methods in physical human–robot collaboration (pHRC) are largely limited to binary soft/hard object classification, hindering fine-grained, safety-critical interaction. Method: We propose an ontology-aware multi-class contact recognition framework distinguishing three categories—human, soft object, and hard object—using time-series force and position data from a Franka Emika Panda manipulator. We design and comparatively evaluate three end-to-end temporal architectures—LSTM, GRU, and Transformer—and rigorously assess the impact of sliding-window preprocessing. Contribution/Results: The optimized Transformer achieves 91.11% accuracy in real-time testing, substantially outperforming conventional binary classification. To our knowledge, this is the first work enabling ontology-aware, three-way contact semantic parsing in pHRC, establishing a deployable foundation for adaptive, safety-aware control in dynamic environments.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Humans and AI: Interaction Techniques and Devices
• Planning, Routing, and Scheduling: Activity and Plan Recognition

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

In physical human-robot collaboration (pHRC) settings, humans and robots collaborate directly in shared environments. Robots must analyze interactions with objects to ensure safety and facilitate meaningful workflows. One critical aspect is human/object detection, where the contacted object is identified. Past research introduced binary machine learning classifiers to distinguish between soft and hard objects. This study improves upon those results by evaluating three-class human/object detection models, offering more detailed contact analysis. A dataset was collected using the Franka Emika Panda robot manipulator, exploring preprocessing strategies for time-series analysis. Models including LSTM, GRU, and Transformers were trained on these datasets. The best-performing model achieved 91.11% accuracy during real-time testing, demonstrating the feasibility of multi-class detection models. Additionally, a comparison of preprocessing strategies suggests a sliding window approach is optimal for this task.

Problem

• Detect multi-class human/object contacts for robot safety
• Improve binary classifiers with three-class detection models
• Evaluate preprocessing strategies for time-series contact analysis

Innovation

• Three-class human/object detection models
• LSTM, GRU, and Transformers for analysis
• Sliding window preprocessing strategy optimal