SIGNET: Motion-Level Knowledge Transfer for Cross-Language Sign Language Translation

2026-06-26

AI Summary

This work addresses the challenges of high spatiotemporal complexity, long sequences, and limited cross-lingual transferability in sign language translation by proposing SIGNET, a novel framework that achieves knowledge transfer across sign languages at the action level for the first time. SIGNET dynamically integrates multiple pretrained expert models through a hand-prior attention mechanism and effectively combines their representations using a gated fusion network. The framework attains state-of-the-art performance on four sign language translation benchmarks—How2Sign, Phoenix14T, CSL-Daily, and MeineDGS—and surpasses existing methods on the WLASL sign language recognition task, demonstrating significantly enhanced generalization and cross-lingual transfer capabilities.

Abstract

Sign language translation (SLT) remains challenging due to its high spatio-temporal complexity, long sequences, and the need to model multiple articulators without relying on gloss annotations. Existing approaches are typically tailored to individual datasets or languages and struggle to scale, while overlooking the relationships between sign languages that could inform more effective cross-lingual transfer. We present \textbf{SIGNET}, a framework that enables motion-level knowledge transfer for cross-language sign language translation. Our key insight is that, although sign languages differ in grammar and lexicon, pretrained models capture motion-level visual patterns that can be reused across datasets and languages. \textbf{SIGNET} integrates multiple pretrained sign language backbones through an attention-based, hand-prior aggregation mechanism that guides a gated fusion network in dynamically selecting the most relevant experts. Comprehensive experiments on four benchmarks (How2Sign, Phoenix14T, CSL-Daily, and MeineDGS) demonstrate state-of-the-art translation performance, and \textbf{SIGNET} also surpasses prior methods on WLASL for sign language recognition.

Problem

• sign language translation
• cross-language transfer
• spatio-temporal complexity
• gloss-free modeling
• multilingual sign languages

Innovation

• motion-level knowledge transfer
• cross-language sign language translation
• attention-based aggregation
• gated fusion network
• pretrained sign language backbones