SignDPO: Multi-level Direct Preference Optimisation for Skeleton-based Gloss-free Sign Language Translation

2026-04-20

AI Summary

Existing vocabulary-free sign language translation models rely on maximum likelihood estimation, which often leads to semantic drift due to insufficient modeling of fine-grained spatiotemporal semantic distinctions. To address this, this work proposes SignDPO, a multi-level direct preference optimization framework that shifts the learning objective from sequence-level imitation to structured preference alignment across spatial, temporal, and linguistic dimensions. SignDPO enables multi-granularity preference learning through hierarchical perturbation for generating suboptimal samples, a self-guided mechanism based on decoder cross-attention, and automatic construction of language-level preferences without manual annotations. Evaluated on CSL-Daily, How2Sign, and OpenASL benchmarks, SignDPO significantly outperforms current vocabulary-free approaches and achieves performance comparable to vocabulary-based models, demonstrating the effectiveness of multi-level preference alignment in sign language translation.

Technology Category

• Natural Language Processing: Learning & Optimization for NLP
• Computer Vision: Language and Vision
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

We present SignDPO, a novel multi-level Direct Preference Optimisation (DPO) framework designed to enhance the alignment of skeleton-based Sign Language Translation. While current skeleton-based models have made significant progress using Maximum Likelihood Estimation, they are primarily constrained by an imitation-based paradigm that lacks discriminative sensitivity to the fine-grained spatio-temporal nuances of sign language, often leading to semantic drift. To address this, SignDPO shifts the optimisation goal from simple sequence mimicry to structured preference alignment across spatial, temporal, and linguistic dimensions. Our framework involves three key designs. First, we introduce a hierarchical perturbation strategy to construct spatial and temporal non-preferred samples at both global and local granularities automatically. Second, we propose a self-guiding mechanism that leverages decoder cross-attention scores to identify and perturb semantically salient skeletal regions, forcing the model to distinguish genuine sign signals from structural distortions. Third, we establish an automated language-level preference generator by fine-tuning a dedicated perturbation model, capturing complex output-level failure modes without manual annotation. Extensive experiments on three widely adopted benchmarks, CSL-Daily, How2Sign, and OpenASL, demonstrate that SignDPO consistently outperforms state-of-the-art gloss-free methods and even rivals established gloss-based ones. Our results suggest that multi-level preference alignment is a powerful paradigm for bridging the gap between high-entropy skeletal trajectories and discrete linguistic semantics.

Problem

• Sign Language Translation
• Skeleton-based
• Semantic Drift
• Preference Alignment
• Gloss-free

Innovation

• Direct Preference Optimisation
• Skeleton-based Sign Language Translation
• Multi-level Alignment
• Self-guiding Mechanism
• Gloss-free Translation