Continuous Bangla Sign Language Translation: Mitigating the Expense of Gloss Annotation with the Assistance of Graph

2025-08-14

AI Summary

To address communication barriers faced by deaf and hard-of-hearing individuals, this paper proposes a novel continuous Bangla Sign Language (BSL) translation approach with weak reliance on gloss annotations. The method innovatively integrates Spatio-Temporal Graph Convolutional Networks (STGCN) with LSTM to model the spatiotemporal dynamics of sign articulation, and couples this with a Transformer architecture to enable end-to-end, lexicon-free translation—thereby significantly reducing dependence on costly, fine-grained gloss-labeled data. Evaluated on RWTH-PHOENIX-2014T, CSL-Daily, and How2Sign, the method achieves BLEU-4 improvements of 4.01, 2.07, and 0.5, respectively, surpassing state-of-the-art approaches. Furthermore, the authors introduce and publicly release BornilDB v1.0—the first dedicated benchmark dataset for continuous BSL translation—establishing its inaugural baseline and advancing low-resource sign language translation research.

Technology Category

• Computer Vision: Language and Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Millions of individuals worldwide are affected by deafness and hearing impairment. Sign language serves as a sophisticated means of communication for the deaf and hard of hearing. However, in societies that prioritize spoken languages, sign language often faces underestimation, leading to communication barriers and social exclusion. The Continuous Bangla Sign Language Translation project aims to address this gap by enhancing translation methods. While recent approaches leverage transformer architecture for state-of-the-art results, our method integrates graph-based methods with the transformer architecture. This fusion, combining transformer and STGCN-LSTM architectures, proves more effective in gloss-free translation. Our contributions include architectural fusion, exploring various fusion strategies, and achieving a new state-of-the-art performance on diverse sign language datasets, namely RWTH-PHOENIX-2014T, CSL-Daily, How2Sign, and BornilDB v1.0. Our approach demonstrates superior performance compared to current translation outcomes across all datasets, showcasing notable improvements of BLEU-4 scores of 4.01, 2.07, and 0.5, surpassing those of GASLT, GASLT and slt_how2sign in RWTH-PHOENIX-2014T, CSL-Daily, and How2Sign, respectively. Also, we introduce benchmarking on the BornilDB v1.0 dataset for the first time. Our method sets a benchmark for future research, emphasizing the importance of gloss-free translation to improve communication accessibility for the deaf and hard of hearing.

Problem

• Developing gloss-free Bangla sign language translation methods
• Integrating graph-based and transformer architectures for better performance
• Improving communication accessibility for the deaf and hard of hearing

Innovation

• Combines transformer and STGCN-LSTM architectures
• Achieves gloss-free sign language translation
• Sets new benchmarks across multiple datasets