MaDiS: Taming Masked Diffusion Language Models for Sign Language Generation

2026-01-27

AI Summary

This work addresses the limitations of existing autoregressive language models in sign language generation, which suffer from unidirectional context modeling and inefficient sequential inference. To overcome these challenges, the authors propose MaDiS, a masked diffusion-based language model that enables bidirectional context modeling and parallel multi-token generation. Key innovations include a three-stage cross-modal pretraining framework spanning token, latent, and 3D physical spaces; a temporal checkpoint-guided demasking strategy that reduces combinatorial complexity by over 10⁴¹-fold; and learnable gated part-mixture embeddings with an optimized codebook. Experiments demonstrate that MaDiS significantly outperforms current methods on the CSL-Daily, Phoenix-2014T, and How2Sign benchmarks, achieving state-of-the-art results in DTW error, SiBLEU, and SiCLIP metrics while reducing inference latency by nearly 30%.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Multimodal Learning
• Computer Vision: Language and Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Sign language generation (SLG) aims to translate written texts into expressive sign motions, bridging communication barriers for the Deaf and Hard-of-Hearing communities. Recent studies formulate SLG within the language modeling framework using autoregressive language models, which suffer from unidirectional context modeling and slow token-by-token inference. To address these limitations, we present MaDiS, a masked-diffusion-based language model for SLG that captures bidirectional dependencies and supports efficient parallel multi-token generation. We further introduce a tri-level cross-modal pretraining scheme that jointly learns from token-, latent-, and 3D physical-space objectives, leading to richer and more grounded sign representations. To accelerate model convergence in the fine-tuning stage, we design a novel unmasking strategy with temporal checkpoints, reducing the combinatorial complexity of unmasking orders by over $10^{41}$ times. In addition, a mixture-of-parts embedding layer is developed to effectively fuse information stored in different part-wise sign tokens through learnable gates and well-optimized codebooks. Extensive experiments on CSL-Daily, Phoenix-2014T, and How2Sign demonstrate that MaDiS achieves superior performance across multiple metrics, including DTW error and two newly introduced metrics, SiBLEU and SiCLIP, while reducing inference latency by nearly 30%. Code and models will be released on our project page.

Problem

• Sign Language Generation
• Masked Diffusion
• Bidirectional Context
• Parallel Generation
• Autoregressive Models

Innovation

• masked diffusion
• sign language generation
• bidirectional modeling
• cross-modal pretraining
• parallel generation