VTaMo: Video-Text Alignment Model for Sign Language Translation

2026-07-10

AI Summary

This work addresses the challenge of insufficient explicit multi-granularity alignment between video and text in morpheme-free sign language translation. To this end, the authors propose the VTaMo framework, which achieves cross-modal alignment at local, global, and positional levels. The method innovatively integrates entropy-regularized optimal transport with learnable null tokens, Earth Mover’s Distance–based orthogonal embedding space calibration, and position-aware contrastive learning to enhance consistency between visual and linguistic representations. Extensive experiments demonstrate that VTaMo achieves state-of-the-art performance across multiple benchmarks, including Phoenix-2014T, CSL-Daily, How2Sign, and OpenASL. Ablation studies further confirm the effectiveness of each proposed component.

Abstract

Sign language translation (SLT) converts continuous sign videos into spoken language text. Gloss-free approaches leverage pre-trained visual encoders and language models but rely on implicit cross-modal alignment from translation supervision alone. We present VTaMo, a framework that introduces explicit multi-granularity alignment at three levels: (1) local alignment via entropy-regularized optimal transport with a learnable null token for fine-grained frame-to-token correspondences; (2) global alignment via a learnable orthogonal transformation that calibrates embedding space geometry through Earth Mover's Distance; and (3) position-aligned contrastive learning for discriminative token-level representations. Experiments on Phoenix-2014T, CSL-Daily, How2Sign, and OpenASL demonstrate consistent state-of-the-art performance, with ablations confirming the complementary contributions of each component. Code is available at https://github.com/junyi2005/vtamo.

Problem

• sign language translation
• video-text alignment
• cross-modal alignment
• multi-granularity alignment
• gloss-free SLT

Innovation

• video-text alignment
• optimal transport
• orthogonal transformation
• contrastive learning
• sign language translation