Towards Skeletal and Signer Noise Reduction in Sign Language Production via Quaternion-Based Pose Encoding and Contrastive Learning

2025-08-20

AI Summary

Neural sign language production (SLP) suffers from high intra-class variability in sign movements and is susceptible to noise arising from signer morphology and stylistic idiosyncrasies. To address these challenges, we propose a robust SLP framework: first, we represent skeletal rotations using quaternions and enforce manifold-aware modeling of pose dynamics via a geodesic loss; second, we introduce a semantics-guided contrastive learning mechanism that explicitly disentangles semantic motion representations from signer-specific nuisance factors in the latent space; third, both components are integrated into a Progressive Transformers architecture. Evaluated on Phoenix14T, our method achieves a 16% improvement in keypoint accuracy and a 6% reduction in mean bone angle error, demonstrating substantial gains in cross-signer generalization and semantic fidelity of generated signs.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Neuro-Symbolic Learning
• Computer Vision: Language and Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

One of the main challenges in neural sign language production (SLP) lies in the high intra-class variability of signs, arising from signer morphology and stylistic variety in the training data. To improve robustness to such variations, we propose two enhancements to the standard Progressive Transformers (PT) architecture (Saunders et al., 2020). First, we encode poses using bone rotations in quaternion space and train with a geodesic loss to improve the accuracy and clarity of angular joint movements. Second, we introduce a contrastive loss to structure decoder embeddings by semantic similarity, using either gloss overlap or SBERT-based sentence similarity, aiming to filter out anatomical and stylistic features that do not convey relevant semantic information. On the Phoenix14T dataset, the contrastive loss alone yields a 16% improvement in Probability of Correct Keypoint over the PT baseline. When combined with quaternion-based pose encoding, the model achieves a 6% reduction in Mean Bone Angle Error. These results point to the benefit of incorporating skeletal structure modeling and semantically guided contrastive objectives on sign pose representations into the training of Transformer-based SLP models.

Problem

• Reducing signer noise and skeletal variability in sign language production
• Improving robustness to anatomical and stylistic variations in signs
• Enhancing accuracy of joint movements and semantic clarity

Innovation

• Quaternion-based pose encoding for angular accuracy
• Contrastive learning for semantic similarity structuring
• Combined geodesic and contrastive losses enhancement