Multimodal Smart Glove for Sign Language Recognition Using Deep Learning

2026-07-08

AI Summary

This study addresses the challenge of achieving both high accuracy and practicality in sign language recognition systems under real-world conditions, which has long hindered effective communication between deaf and hearing individuals. The authors propose a wearable smart glove based on multimodal fusion that simultaneously captures finger flexion, inertial measurement unit (IMU) data, and facial visual cues. A lightweight LSTM network is designed to model the temporal dynamics of gestures, marking the first integration of multimodal hand and facial expression perception with embedded deep learning. The system deploys a TensorFlow Lite model on the ESP32-C6 platform, enabling end-to-end real-time sign language translation. Experimental results demonstrate an overall recognition accuracy of 95%, confirming the system’s high precision, low latency, and feasibility for real-world deployment.

Abstract

Sign language recognition technologies can improve communication between deaf individuals and the broader community, but many existing systems face challenges in real-world deployment. This paper presents a deployable smart glove system for sign language recognition that integrates wearable sensing and deep learning. The glove incorporates flex sensors and an inertial measurement unit (IMU) to capture finger articulation and hand motion, while facial cues are obtained through a camera. Sensor data are transmitted via an ESP32-C6 microcontroller and processed using a long short-term memory (LSTM) network to model temporal gesture dynamics. Experimental results show that the proposed model achieves an overall recognition accuracy of approximately 95%. The trained model is further converted to TensorFlow Lite for real-time inference. This demonstrates the feasibility of the system for practical sign language translation applications.

Problem

• sign language recognition
• real-world deployment
• wearable sensing
• deep learning
• real-time inference

Innovation

• multimodal sensing
• smart glove
• sign language recognition
• LSTM
• TensorFlow Lite