Sign language recognition from skeletal data using graph and recurrent neural networks

2025-11-08

AI Summary

This work addresses isolated sign language gesture recognition by proposing a graph-temporal joint modeling approach based on skeletal pose sequences. To simultaneously capture the spatial topological structure among joints and the dynamic temporal evolution of gestures, we design Graph-GRU: a hybrid architecture integrating graph neural networks (GNNs) to model spatial dependencies in the skeleton topology and gated recurrent units (GRUs) to encode long-range temporal dynamics. The framework enables end-to-end learning of pose-driven spatiotemporal feature representations and exhibits strong scalability. Extensive experiments on the large-scale AUTSL sign language dataset demonstrate significant improvements in classification accuracy over state-of-the-art methods. Results validate the effectiveness of jointly leveraging structural priors (via graph-based spatial modeling) and sequential dynamics (via recurrent temporal modeling) for enhancing sign language recognition performance. This work establishes a novel paradigm for pose-based sign language understanding.

Technology Category

• Machine Learning: Graph-based Machine Learning
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Computer Vision: Biometrics, Face, Gesture & Pose

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding

Abstract

This work presents an approach for recognizing isolated sign language gestures using skeleton-based pose data extracted from video sequences. A Graph-GRU temporal network is proposed to model both spatial and temporal dependencies between frames, enabling accurate classification. The model is trained and evaluated on the AUTSL (Ankara university Turkish sign language) dataset, achieving high accuracy. Experimental results demonstrate the effectiveness of integrating graph-based spatial representations with temporal modeling, providing a scalable framework for sign language recognition. The results of this approach highlight the potential of pose-driven methods for sign language understanding.

Problem

• Recognizing isolated sign language gestures from skeletal pose data
• Modeling spatial and temporal dependencies for accurate classification
• Providing a scalable framework for sign language understanding

Innovation

• Graph-GRU network models spatial-temporal dependencies
• Skeleton-based pose data extracted from video sequences
• Graph-based spatial representations integrated with temporal modeling