ViPo-MLLM: Visual-Pose Multimodal LLM for Gloss-Free Sign Language Translation

2026-07-03

AI Summary

This work addresses the challenging task of translating sign language videos into spoken language sentences without gloss-level supervision. The authors propose a multimodal fusion approach that jointly models RGB video frames and human pose sequences. Dedicated spatiotemporal encoders extract dynamic visual and pose features, while a cross-modal attention mechanism captures long-range dependencies between modalities. The framework further integrates structured prompts with a large language model, enabling end-to-end training under combined contrastive learning and language modeling objectives. This method represents the first effective fusion of visual and pose information in a gloss-free setting, achieving state-of-the-art performance on the PHOENIX14T and CSL-Daily benchmarks—surpassing prior gloss-free approaches and rivaling advanced methods that rely on gloss supervision.

Abstract

Gloss-free Sign Language Translation (SLT) translates sign language videos into spoken-language sentences without gloss annotations, avoiding costly labeling but requiring fine-grained modeling of hands, body, and facial cues. Existing methods often use single-modality or weakly fused features, limiting performance. We propose ViPo-MLLM, a framework that integrates spatio-temporal RGB and human pose features. Dedicated encoders model intra-modal dynamics and cross-modal attention captures long-range dependencies. The fused representation is conditioned with a structured prompt and processed by an LLM trained with contrastive and language modeling objectives. The proposed model was evaluated on the PHOENIX14T and CSL-Daily datasets and achieved new state-of-the-art results on both datasets. Moreover, the ViPo-MLLM model attained competitive performance compared to gloss-based recognition approaches, confirming the effectiveness of the proposed pose cues and cross-modal attention mechanisms.

Problem

• Gloss-free Sign Language Translation
• Multimodal Learning
• Human Pose
• Visual-Pose Fusion
• Sign Language Understanding

Innovation

• Multimodal LLM
• Pose-guided representation
• Gloss-free Sign Language Translation
• Cross-modal attention
• Structured prompting