HandReader: Advanced Techniques for Efficient Fingerspelling Recognition

2025-05-15

AI Summary

This work addresses three key challenges in sign language fingerspelling recognition: difficulty modeling rapid hand motion, weak handling of variable-length video sequences, and low accuracy in recognizing proper nouns. To tackle these, we propose: (1) a Temporal Shift Adaptive Module (TSAM) and a Temporal Pose Encoder (TPE) for joint temporal modeling of RGB and keypoint features; (2) the first RGB-keypoint multimodal fusion architecture, integrating 2D/3D convolutions, temporal feature alignment, and keypoint tensor representation; and (3) Znaki—the first publicly available Russian fingerspelling dataset, which we construct and open-source. Our method achieves state-of-the-art performance on ChicagoFSWild, ChicagoFSWild+, and Znaki. Both the model and the Znaki dataset are publicly released to foster further research.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs

Abstract

Fingerspelling is a significant component of Sign Language (SL), allowing the interpretation of proper names, characterized by fast hand movements during signing. Although previous works on fingerspelling recognition have focused on processing the temporal dimension of videos, there remains room for improving the accuracy of these approaches. This paper introduces HandReader, a group of three architectures designed to address the fingerspelling recognition task. HandReader$_{RGB}$ employs the novel Temporal Shift-Adaptive Module (TSAM) to process RGB features from videos of varying lengths while preserving important sequential information. HandReader$_{KP}$ is built on the proposed Temporal Pose Encoder (TPE) operated on keypoints as tensors. Such keypoints composition in a batch allows the encoder to pass them through 2D and 3D convolution layers, utilizing temporal and spatial information and accumulating keypoints coordinates. We also introduce HandReader_RGB+KP - architecture with a joint encoder to benefit from RGB and keypoint modalities. Each HandReader model possesses distinct advantages and achieves state-of-the-art results on the ChicagoFSWild and ChicagoFSWild+ datasets. Moreover, the models demonstrate high performance on the first open dataset for Russian fingerspelling, Znaki, presented in this paper. The Znaki dataset and HandReader pre-trained models are publicly available.

Problem

• Improving accuracy in fingerspelling recognition from videos
• Processing temporal and spatial information in sign language
• Introducing a dataset for Russian fingerspelling recognition

Innovation

• Uses Temporal Shift-Adaptive Module for RGB
• Employs Temporal Pose Encoder for keypoints
• Combines RGB and keypoints with joint encoder