Deep Learning-Based Sign Language Recognition from Videos and Cross-Lingual Translation to Indian Vernaculars

2026-06-21

AI Summary

This work addresses the scarcity of automatic sign language recognition and translation tools for low-resource Indian languages by proposing a two-stage deep learning framework. First, a fine-tuned VideoMAE video Transformer classifies short sign language videos into 13 English lexical categories; subsequently, Meta AI’s NLLB-200 multilingual machine translation model renders these outputs into Hindi, Telugu, and Bengali. This approach represents the first integration of video Transformers with state-of-the-art multilingual neural translation systems to achieve end-to-end sign language understanding tailored to Indian languages. Evaluated on the AI4Bharat Sign Language Dataset, the model attains 99% training accuracy and 78% validation accuracy. The authors publicly release the source code and deploy an interactive Streamlit demo platform enabling users to upload sign language videos and receive real-time multilingual translations.

Abstract

Sign language is a primary mode of communication for the global deaf and hard-of-hearing community, yet automated tools that recognize sign gestures from video and translate them into natural language text remain limited, particularly for low-resource Indian languages. We present a two-stage deep learning pipeline that (i) classifies short sign language video clips into English word labels using a fine-tuned VideoMAE video transformer, and (ii) translates the predicted English label into Hindi, Telugu, and Bengali using Meta AI's No Language Left Behind (NLLB-200) multilingual translation model. The classification model is fine-tuned on a 13-class subset of the AI4Bharat Indian Sign Language video corpus from IIT Madras, processing 16-frame clips sampled uniformly from each video at 224 x 224 resolution. Under a small-scale academic setting (13 classes, 197 clips, 80-20 split), the fine-tuned model reaches 99% training accuracy and 78% validation accuracy after 15 epochs. We provide a per-class breakdown via a confusion matrix and classification report, identify the dominant failure modes (confusable adjective pairs such as ugly, deaf, blind, hat, and dress), and describe a Streamlit-based inference demo that takes a user-uploaded video and returns the predicted English label alongside its Hindi, Telugu, and Bengali translations. We discuss the scope, limitations (small label set, isolated-word rather than continuous signing, single-signer style sensitivity, ambiguity of single-word machine translation), and directions for future work, including expanding to sentence-level generation and a larger vocabulary. Code is released to support reproducibility.

Problem

• Sign Language Recognition
• Cross-Lingual Translation
• Low-Resource Languages
• Video-based Gesture Recognition
• Indian Vernaculars

Innovation

• VideoMAE
• sign language recognition
• cross-lingual translation
• NLLB-200
• Indian Sign Language