Q-BridgeNet: A Quantization Network for Cross-Lingual Sign Language Translation

2026-07-13

AI Summary

This work addresses the challenge of balancing semantic sharing and language-specific characteristics within a unified model for cross-lingual sign language translation, a problem that often leads to inter-lingual interference. The authors propose Q-BridgeNet, a novel end-to-end framework that introduces adaptive segmentation and residual vector quantization on the sign language side to generate discrete Q-unit representations, while fine-tuning a multilingual large language model on the spoken language side. By innovatively integrating a shared base codebook with language-specific residual codebooks, the method effectively disentangles linguistic commonalities from idiosyncrasies. Evaluated on PHOENIX14T, How2Sign, and CSL-Daily benchmarks, Q-BridgeNet achieves state-of-the-art performance, substantially mitigating cross-lingual conflict and demonstrating strong generalization capabilities even on non-native language pairs.

Abstract

Most sign language translation (SLT) methods focus on isolated native sign-spoken pairs (e.g., American Sign Language - English). Extending language-specific SLT models to multilingual translation would improve accessibility by enabling communication across diverse sign and spoken language communities. However, existing multilingual SLT approaches still struggle to learn a unified model that minimizes cross-lingual conflicts while capturing shared cross-lingual semantics and preserving language-specific variations across different sign languages. Therefore, we propose Q-BridgeNet, a unified framework for multilingual SLT that jointly mitigates cross-lingual conflicts across both the sign language and spoken language sides. On the sign language side, Q-BridgeNet learns discrete Q-units via adaptive segmentation and residual vector quantization: a shared base codebook provides language-agnostic semantic primitives, while language-specific residual codebooks refine heterogeneous signing semantics. On the spoken language side, a multilingual LLM is fine-tuned to operate in the Q-unit space, leveraging cross-lingual priors to enable a unified SLT model. Experiments on PHOENIX14T, How2Sign, and CSL-Daily show that Q-BridgeNet effectively mitigates cross-lingual conflicts, achieving state-of-the-art performance on native sign-spoken pairs while also demonstrating strong generalization to non-native pairs. Our source code is publicly available at: https://github.com/FengLiQ/Q-BridgeNet

Problem

• cross-lingual sign language translation
• multilingual SLT
• cross-lingual conflicts
• language-specific variations
• shared semantics

Innovation

• Quantization Network
• Cross-Lingual Sign Language Translation
• Residual Vector Quantization
• Multilingual LLM
• Q-units