Leveraging Large Language Models for Accurate Sign Language Translation in Low-Resource Scenarios

2025-08-25

AI Summary

Natural language-to-sign language (NL2SL) translation suffers from severe low-resource constraints—specifically, the absence of large-scale, standardized, and domain-balanced parallel corpora—leading to poor generalization in existing models. To address this, we propose AulSign, the first framework integrating dynamic prompt engineering, in-context learning, and symbolic gesture association. Without relying on built-in sign language knowledge, AulSign models gesture semantics solely through natural language descriptions and introduces a sample selection mechanism to enhance large language models’ implicit understanding of sign language structure. This establishes a scalable, multilingual NL2SL translation paradigm. On SignBank+ and LaCAM CNR-ISTC benchmarks, AulSign significantly outperforms prior state-of-the-art methods, demonstrating high accuracy and strong generalization under low-resource conditions. Our approach opens a new pathway toward accessible human–machine interaction.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Neuro-Symbolic Learning
• Computer Vision: Language and Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Translating natural languages into sign languages is a highly complex and underexplored task. Despite growing interest in accessibility and inclusivity, the development of robust translation systems remains hindered by the limited availability of parallel corpora which align natural language with sign language data. Existing methods often struggle to generalize in these data-scarce environments, as the few datasets available are typically domain-specific, lack standardization, or fail to capture the full linguistic richness of sign languages. To address this limitation, we propose Advanced Use of LLMs for Sign Language Translation (AulSign), a novel method that leverages Large Language Models via dynamic prompting and in-context learning with sample selection and subsequent sign association. Despite their impressive abilities in processing text, LLMs lack intrinsic knowledge of sign languages; therefore, they are unable to natively perform this kind of translation. To overcome this limitation, we associate the signs with compact descriptions in natural language and instruct the model to use them. We evaluate our method on both English and Italian languages using SignBank+, a recognized benchmark in the field, as well as the Italian LaCAM CNR-ISTC dataset. We demonstrate superior performance compared to state-of-the-art models in low-data scenario. Our findings demonstrate the effectiveness of AulSign, with the potential to enhance accessibility and inclusivity in communication technologies for underrepresented linguistic communities.

Problem

• Addressing sign language translation in low-resource data scenarios
• Overcoming limited parallel corpora for natural-sign language alignment
• Enhancing generalization in data-scarce sign language translation environments

Innovation

• Leverages LLMs with dynamic prompting
• Uses in-context learning with sample selection
• Associates signs with natural language descriptions