SignLlama: Enhancing Gloss-free Sign Language Translation by Prioritizing Visual Features for LLMs

2026-08-09

AI Summary

This work addresses the challenge that large language models struggle to effectively leverage visual information in gloss-free sign language translation and tend to over-rely on textual features during multimodal fusion. To mitigate this, the authors propose a pseudo-gloss CTC pretraining strategy coupled with a vision-priority knowledge distillation mechanism. The former enhances the visual backbone’s capacity to capture visual semantics, while the latter enforces the model to establish a prediction pathway relying solely on visual input, thereby bridging the distributional gap between visual and textual features. Built upon a unified large language model architecture, the proposed approach achieves state-of-the-art performance across multiple gloss-free sign language translation benchmarks without requiring additional modalities or external sign language data for pretraining.

Abstract

Large Language Models (LLMs) have achieved remarkable success across a wide range of tasks. However, fine-tuning LLMs for Gloss-Free Sign Language Translation (GFSLT) remains a challenge. In this paper, we investigate how to effectively adapt LLMs to the GFSLT task. We show that there are two key issues that need to be solved: (1) the inherent distributional gap between visual feature inputs and text feature inputs makes it difficult for LLMs to interpret visual inputs; and (2) existing approaches typically concatenate visual and textual features in an autoregressive framework, which leads to the model overemphasizing textual inputs and deprioritizing visual cues, as LLMs are pretrained predominantly on text-centric data. To address the first challenge, we propose a simple yet effective method named Filtered Pseudo-Gloss CTC Pretraining, which leverages filtered pseudo-gloss sequences generated from text sequences to supervise the training of the visual backbone. To tackle the second issue, we introduce a Visual-Prioritized Distillation training strategy. Specifically, we define a visual-only prediction path in which text inputs are masked, and the model is required to generate the target sequence relying solely on visual inputs. To guide this path, the outputs from the standard visual-textual prediction are then distilled into the visual-only prediction path, encouraging the model to prioritize visual features. Comprehensive experiments and qualitative analyses demonstrate the effectiveness of the proposed model. The proposed SignLlama achieves very competitive performance on multiple datasets for GFSLT tasks, without using any extra modalities or external sign language datasets for pretraining.

Problem

• Gloss-Free Sign Language Translation
• Large Language Models
• Visual Features
• Distributional Gap
• Visual-Textual Integration

Innovation

• Gloss-Free Sign Language Translation
• Visual-Prioritized Distillation
• Filtered Pseudo-Gloss CTC Pretraining
• Large Language Models
• Multimodal Learning