A Bimanual Gesture Interface for ROS-Based Mobile Manipulators Using TinyML and Sensor Fusion

2025-09-23

AI Summary

To address low reliability, poor efficiency, and unintuitive interaction in gesture-based control of mobile manipulators, this paper proposes a dual-hand collaborative control interface leveraging TinyML and multi-sensor fusion. The method fuses accelerometer, bend sensor, and IMU data, extracts spectral features, and deploys a lightweight neural network for low-power, real-time gesture recognition on-device. An adaptive sensor fusion algorithm enhances robustness, while deep ROS integration enables synchronized execution of mobile base navigation and 7-DOF Kinova Gen3 manipulator control. Key contributions include the first open-source, deployable dual-hand collaborative architecture specifically designed for mobile manipulators—significantly improving natural interaction and task parallelism. Experimental results demonstrate >96% recognition accuracy, average latency <80 ms, and power consumption below 120 mW.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Biometrics, Face, Gesture & Pose
• Planning, Routing, and Scheduling: Activity and Plan Recognition

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Gesture-based control for mobile manipulators faces persistent challenges in reliability, efficiency, and intuitiveness. This paper presents a dual-hand gesture interface that integrates TinyML, spectral analysis, and sensor fusion within a ROS framework to address these limitations. The system uses left-hand tilt and finger flexion, captured using accelerometer and flex sensors, for mobile base navigation, while right-hand IMU signals are processed through spectral analysis and classified by a lightweight neural network. This pipeline enables TinyML-based gesture recognition to control a 7-DOF Kinova Gen3 manipulator. By supporting simultaneous navigation and manipulation, the framework improves efficiency and coordination compared to sequential methods. Key contributions include a bimanual control architecture, real-time low-power gesture recognition, robust multimodal sensor fusion, and a scalable ROS-based implementation. The proposed approach advances Human-Robot Interaction (HRI) for industrial automation, assistive robotics, and hazardous environments, offering a cost-effective, open-source solution with strong potential for real-world deployment and further optimization.

Problem

• Developing reliable gesture control for mobile manipulators using dual-hand inputs
• Enabling simultaneous navigation and manipulation through sensor fusion techniques
• Creating efficient real-time gesture recognition with low-power TinyML implementation

Innovation

• Bimanual gesture interface with TinyML integration
• Sensor fusion combining accelerometer and IMU data
• ROS-based framework enabling simultaneous navigation manipulation