Improving Handshape Representations for Sign Language Processing: A Graph Neural Network Approach

2025-09-22

AI Summary

Existing sign language processing models lack explicit, structured representations of handshape, limiting both recognition accuracy and linguistic analysis capabilities. Method: We introduce the first structured handshape recognition benchmark for sign language sequences and propose a decoupled spatiotemporal modeling framework: an anatomy-guided graph neural network encodes static handshape topology, while contrastive learning captures dynamic handshape evolution over time. This design explicitly disentangles intrinsic handshape morphology from temporal motion patterns, enhancing representation discriminability. Contribution/Results: Our method achieves 46% accuracy on a 37-class handshape recognition task—surpassing the strongest baseline by 21 percentage points (+21%). These results empirically validate that structured, decoupled modeling is essential for advancing sign language understanding, particularly in bridging low-level visual perception with higher-level linguistic interpretation.

Technology Category

• Machine Learning: Structured Learning
• Computer Vision: Language and Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding

Abstract

Handshapes serve a fundamental phonological role in signed languages, with American Sign Language employing approximately 50 distinct shapes. However,computational approaches rarely model handshapes explicitly, limiting both recognition accuracy and linguistic analysis.We introduce a novel graph neural network that separates temporal dynamics from static handshape configurations. Our approach combines anatomically-informed graph structures with contrastive learning to address key challenges in handshape recognition, including subtle interclass distinctions and temporal variations. We establish the first benchmark for structured handshape recognition in signing sequences, achieving 46% accuracy across 37 handshape classes (with baseline methods achieving 25%).

Problem

• Modeling handshape configurations explicitly for sign language processing
• Addressing subtle interclass distinctions and temporal variations in recognition
• Establishing benchmark for structured handshape recognition in signing sequences

Innovation

• Graph neural network separates temporal dynamics
• Anatomically-informed graph structures with contrastive learning
• Establishes benchmark for structured handshape recognition