Think in Latent Thoughts: A New Paradigm for Gloss-Free Sign Language Translation

2026-04-16

AI Summary

This work addresses a critical limitation in existing sign language translation methods, which erroneously map isolated sign segments directly to spoken-language words while neglecting the contextual, spatial, and dynamic nature of sign language semantics. To overcome this, the authors propose a reasoning-driven cross-modal translation framework that introduces an ordered latent chain-of-thought sequence as an intermediate representation between video and text. The framework employs a “plan-then-align” decoding mechanism that decouples semantic planning from evidence grounding, enabling more coherent and faithful translation. Additionally, the study presents the first large-scale, gloss-free sign language video–text parallel dataset with strong contextual dependencies. Experimental results demonstrate that the proposed approach significantly outperforms current gloss-free methods across multiple benchmarks, achieving notable improvements in semantic coherence and translation fidelity.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Language and Vision
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Many SLT systems quietly assume that brief chunks of signing map directly to spoken-language words. That assumption breaks down because signers often create meaning on the fly using context, space, and movement. We revisit SLT and argue that it is mainly a cross-modal reasoning task, not just a straightforward video-to-text conversion. We thus introduce a reasoning-driven SLT framework that uses an ordered sequence of latent thoughts as an explicit middle layer between the video and the generated text. These latent thoughts gradually extract and organize meaning over time. On top of this, we use a plan-then-ground decoding method: the model first decides what it wants to say, and then looks back at the video to find the evidence. This separation improves coherence and faithfulness. We also built and released a new large-scale gloss-free SLT dataset with stronger context dependencies and more realistic meanings. Experiments across several benchmarks show consistent gains over existing gloss-free methods. Code and data will be released upon acceptance at https://github.com/fletcherjiang/SignThought.

Problem

• Sign Language Translation
• Cross-modal Reasoning
• Gloss-free
• Context Dependency
• Latent Meaning

Innovation

• latent thoughts
• cross-modal reasoning
• gloss-free sign language translation
• plan-then-ground decoding
• context-aware SLT dataset