Skin-Machine Interface with Multimodal Contact Motion Classifier

2025-07-25

AI Summary

Complex robotic manipulation demands natural, robust human–robot interaction in dynamic, unstructured environments. Method: We propose a multimodal human–robot interface based on flexible skin-mounted sensors, integrating triaxial acceleration and tactile signals. A lightweight recurrent neural network (RNN) architecture is developed for time-series classification, systematically investigating the impact of sensor modality fusion and flexible mounting configurations on classification robustness and training stability. Contribution/Results: Our work uncovers the intrinsic coupling mechanism among multimodal perception, mechanical compliance, and temporal modeling—enabling significant improvements in contact-motion recognition accuracy. Experimental evaluation demonstrates >95% classification accuracy across diverse contact actions; the interface successfully controls a dual-arm mobile manipulator to perform grasping, navigation, and collaborative tasks. This validates the feasibility and practicality of skin–machine interfaces for high-dynamics, real-world robotic applications.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Humans and AI: Interaction Techniques and Devices
• Machine Learning: Multimodal Learning

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

This paper proposes a novel framework for utilizing skin sensors as a new operation interface of complex robots. The skin sensors employed in this study possess the capability to quantify multimodal tactile information at multiple contact points. The time-series data generated from these sensors is anticipated to facilitate the classification of diverse contact motions exhibited by an operator. By mapping the classification results with robot motion primitives, a diverse range of robot motions can be generated by altering the manner in which the skin sensors are interacted with. In this paper, we focus on a learning-based contact motion classifier employing recurrent neural networks. This classifier is a pivotal factor in the success of this framework. Furthermore, we elucidate the requisite conditions for software-hardware designs. Firstly, multimodal sensing and its comprehensive encoding significantly contribute to the enhancement of classification accuracy and learning stability. Utilizing all modalities simultaneously as inputs to the classifier proves to be an effective approach. Secondly, it is essential to mount the skin sensors on a flexible and compliant support to enable the activation of three-axis accelerometers. These accelerometers are capable of measuring horizontal tactile information, thereby enhancing the correlation with other modalities. Furthermore, they serve to absorb the noises generated by the robot's movements during deployment. Through these discoveries, the accuracy of the developed classifier surpassed 95 %, enabling the dual-arm mobile manipulator to execute a diverse range of tasks via the Skin-Machine Interface. https://youtu.be/UjUXT4Z4BC8

Problem

• Classifying diverse contact motions using skin sensors
• Mapping contact motions to robot movement primitives
• Enhancing accuracy with multimodal sensing and flexible supports

Innovation

• Multimodal skin sensors for tactile data
• Recurrent neural networks for motion classification
• Flexible support for enhanced sensor accuracy