Geo-Sign: Hyperbolic Contrastive Regularisation for Geometrically Aware Sign Language Translation

2025-05-30

AI Summary

This work addresses two key challenges in sign language translation (SLT): geometric distortion in skeletal representations and the difficulty of modeling fine-grained, hierarchical motion—particularly finger articulation. We propose the first end-to-end SLT framework grounded in hyperbolic geometry. Methodologically, we map spatiotemporal skeletal features extracted by ST-GCN onto the Poincaré ball, introducing a differentiable hyperbolic projection layer, weighted Fréchet mean aggregation, and a hyperbolic contrastive loss to explicitly encode the hierarchical dynamical priors inherent in sign language motion. The design ensures privacy preservation (operating solely on skeletal data, without raw video), computational efficiency, and geometric fidelity. Experiments on mainstream SLT benchmarks demonstrate that our approach surpasses state-of-the-art RGB-based methods; notably, it achieves significant gains in finger-level action translation accuracy and accelerates inference by 23%.

Technology Category

• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Language and Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• Security and Privacy: Data transparency and provenance

Abstract

Recent progress in Sign Language Translation (SLT) has focussed primarily on improving the representational capacity of large language models to incorporate Sign Language features. This work explores an alternative direction: enhancing the geometric properties of skeletal representations themselves. We propose Geo-Sign, a method that leverages the properties of hyperbolic geometry to model the hierarchical structure inherent in sign language kinematics. By projecting skeletal features derived from Spatio-Temporal Graph Convolutional Networks (ST-GCNs) into the Poincar'e ball model, we aim to create more discriminative embeddings, particularly for fine-grained motions like finger articulations. We introduce a hyperbolic projection layer, a weighted Fr'echet mean aggregation scheme, and a geometric contrastive loss operating directly in hyperbolic space. These components are integrated into an end-to-end translation framework as a regularisation function, to enhance the representations within the language model. This work demonstrates the potential of hyperbolic geometry to improve skeletal representations for Sign Language Translation, improving on SOTA RGB methods while preserving privacy and improving computational efficiency. Code available here: https://github.com/ed-fish/geo-sign.

Problem

• Enhancing geometric properties of skeletal representations for SLT
• Modeling hierarchical sign language structure via hyperbolic geometry
• Improving discriminative embeddings for fine-grained sign motions

Innovation

• Hyperbolic geometry for skeletal representation enhancement
• Poincaré ball projection for discriminative embeddings
• Geometric contrastive loss in hyperbolic space