Indian Sign Language Detection for Real-Time Translation using Machine Learning

2025-07-27

AI Summary

To address the longstanding scarcity of sign language translation resources for the Deaf and hard-of-hearing community in India, this work proposes a lightweight, end-to-end real-time Indian Sign Language (ISL) recognition and translation framework. Methodologically, it integrates MediaPipe’s robust hand keypoint detection with a customized lightweight CNN classifier to establish a low-latency visual-spatial feature modeling pipeline; dynamic temporal normalization and targeted data augmentation further enhance generalization. Evaluated on the public ISL dataset, our system achieves 99.95% classification accuracy (F1 = 0.999), outperforming all existing baselines—marking the first such result for ISL. It satisfies strict real-time constraints (<30 ms per frame inference) and supports efficient mobile deployment. This work fills a critical gap in high-accuracy, real-time ISL translation technology and provides a scalable technical foundation for inclusive human-computer interaction and societal accessibility.

Technology Category

• Computer Vision: Language and Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Gestural language is used by deaf & mute communities to communicate through hand gestures & body movements that rely on visual-spatial patterns known as sign languages. Sign languages, which rely on visual-spatial patterns of hand gestures & body movements, are the primary mode of communication for deaf & mute communities worldwide. Effective communication is fundamental to human interaction, yet individuals in these communities often face significant barriers due to a scarcity of skilled interpreters & accessible translation technologies. This research specifically addresses these challenges within the Indian context by focusing on Indian Sign Language (ISL). By leveraging machine learning, this study aims to bridge the critical communication gap for the deaf & hard-of-hearing population in India, where technological solutions for ISL are less developed compared to other global sign languages. We propose a robust, real-time ISL detection & translation system built upon a Convolutional Neural Network (CNN). Our model is trained on a comprehensive ISL dataset & demonstrates exceptional performance, achieving a classification accuracy of 99.95%. This high precision underscores the model's capability to discern the nuanced visual features of different signs. The system's effectiveness is rigorously evaluated using key performance metrics, including accuracy, F1 score, precision & recall, ensuring its reliability for real-world applications. For real-time implementation, the framework integrates MediaPipe for precise hand tracking & motion detection, enabling seamless translation of dynamic gestures. This paper provides a detailed account of the model's architecture, the data preprocessing pipeline & the classification methodology. The research elaborates the model architecture, preprocessing & classification methodologies for enhancing communication in deaf & mute communities.

Problem

• Detects Indian Sign Language in real-time using machine learning
• Bridges communication gap for deaf communities with CNN technology
• Achieves 99.95% accuracy in ISL gesture classification

Innovation

• Uses CNN for Indian Sign Language detection
• Integrates MediaPipe for real-time hand tracking
• Achieves 99.95% classification accuracy