Human-Inspired Soft Anthropomorphic Hand System for Neuromorphic Object and Pose Recognition Using Multimodal Signals

2025-09-02

AI Summary

To address the challenges of multimodal sensory fusion and poor cross-pose recognition robustness in soft biomimetic hands, this paper proposes a flexible biomimetic robotic hand inspired by the human somatosensory system, integrating tactile, proprioceptive, and thermal sensing modalities. Methodologically, we design a biologically inspired spike encoding scheme and a novel differential spiking neuron model, coupled with dynamic thermal response modeling and a multi-sensor spatiotemporal alignment fusion strategy, enabling low-power, high-temporal-resolution collaborative processing of heterogeneous signals. Experiments demonstrate that the system achieves 97.14% object recognition accuracy under cross-pose conditions—significantly outperforming existing soft hands—and shows marked improvement in material classification, validating the efficacy of the proposed encoding and neuromorphic computing paradigm. The core contribution lies in the first deep integration of thermal sensing into a multimodal spiking perception framework for soft robotic hands, achieving biologically interpretable and computationally efficient fusion-based recognition.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Computer Vision: Multi-modal Vision

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

The human somatosensory system integrates multimodal sensory feedback, including tactile, proprioceptive, and thermal signals, to enable comprehensive perception and effective interaction with the environment. Inspired by the biological mechanism, we present a sensorized soft anthropomorphic hand equipped with diverse sensors designed to emulate the sensory modalities of the human hand. This system incorporates biologically inspired encoding schemes that convert multimodal sensory data into spike trains, enabling highly-efficient processing through Spiking Neural Networks (SNNs). By utilizing these neuromorphic signals, the proposed framework achieves 97.14% accuracy in object recognition across varying poses, significantly outperforming previous studies on soft hands. Additionally, we introduce a novel differentiator neuron model to enhance material classification by capturing dynamic thermal responses. Our results demonstrate the benefits of multimodal sensory fusion and highlight the potential of neuromorphic approaches for achieving efficient, robust, and human-like perception in robotic systems.

Problem

• Developing a soft robotic hand with multimodal sensors for object recognition
• Converting sensory data into spike trains for efficient neuromorphic processing
• Enhancing material classification using dynamic thermal response differentiation

Innovation

• Soft hand with multimodal sensors mimicking human senses
• Biologically inspired encoding converts data to spike trains
• Novel differentiator neuron model enhances thermal classification