Hierarchical Feature Alignment for Gloss-Free Sign Language Translation

2025-07-09

AI Summary

This work addresses end-to-end sign language translation (SLT) from gloss-free sign videos to spoken-language sentences. Methodologically, it proposes a hierarchical alignment and structure-aware pretraining framework: (1) a frame–segment–video three-level feature alignment mechanism to explicitly model the spatiotemporal structure of sign language; (2) pseudo-gloss-guided multi-granularity contrastive learning to achieve cross-modal semantic alignment between video and text; and (3) integration of large language model (LLM)-generated textual representations to enhance semantic consistency. Its key contribution lies in eliminating explicit gloss supervision by leveraging structure-driven hierarchical pretraining and pseudo-label-based contrastive alignment, thereby bridging the representation gap between visual and linguistic modalities. Experiments demonstrate substantial improvements across multiple benchmarks—+3.2 BLEU-4 and +2.8 ROUGE-L—while maintaining efficient inference, validating both effectiveness and practicality.

Technology Category

• Computer Vision: Language and Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Sign Language Translation (SLT) attempts to convert sign language videos into spoken sentences. However, many existing methods struggle with the disparity between visual and textual representations during end-to-end learning. Gloss-based approaches help to bridge this gap by leveraging structured linguistic information. While, gloss-free methods offer greater flexibility and remove the burden of annotation, they require effective alignment strategies. Recent advances in Large Language Models (LLMs) have enabled gloss-free SLT by generating text-like representations from sign videos. In this work, we introduce a novel hierarchical pre-training strategy inspired by the structure of sign language, incorporating pseudo-glosses and contrastive video-language alignment. Our method hierarchically extracts features at frame, segment, and video levels, aligning them with pseudo-glosses and the spoken sentence to enhance translation quality. Experiments demonstrate that our approach improves BLEU-4 and ROUGE scores while maintaining efficiency.

Problem

• Bridging visual-textual disparity in sign language translation
• Enhancing gloss-free SLT with hierarchical alignment strategies
• Improving translation quality without gloss annotations

Innovation

• Hierarchical pre-training with pseudo-glosses
• Contrastive video-language alignment strategy
• Multi-level feature extraction for SLT