HandyLabel: Towards Post-Processing to Real-Time Annotation Using Skeleton Based Hand Gesture Recognition

2025-11-27

AI Summary

Traditional data labeling relies on post-hoc annotation, which is time-consuming and labor-intensive; moreover, subjective memory biases regarding cognitive states (e.g., emotion, comprehension level) often introduce labeling errors. To address this, we propose HandyLabel—a real-time gesture annotation tool leveraging hand skeletal information. It enables user-defined gesture-to-label mapping and supports interactive, post-processing-free labeling, thereby significantly reducing reliance on memory and minimizing annotation errors. Methodologically, we employ a ResNet50-based model trained and evaluated on the HaGRID dataset using preprocessed skeletal images, and deliver real-time interaction via a web interface. Experiments achieve an F1-score of 0.923; user studies indicate that 88.9% of participants prefer HandyLabel over conventional approaches. This work pioneers the deep integration of skeleton-driven gesture recognition into real-time annotation workflows, substantially improving labeling efficiency, accuracy, and user experience.

Technology Category

• Computer Vision: Biometrics, Face, Gesture & Pose
• Humans and AI: Interaction Techniques and Devices
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Representation, semantic annotation, enhancement, enrichments, access and/or integration of a variety of data on the Web
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

The success of machine learning is deeply linked to the availability of high-quality training data, yet retrieving and manually labeling new data remains a time-consuming and error-prone process. Traditional annotation tools, such as Label Studio, often require post-processing, where users label data after it has been recorded. Post-processing is highly time-consuming and labor-intensive, especially with large datasets, and may lead to erroneous annotations due to the difficulty of subjects' memory tasks when labeling cognitive activities such as emotions or comprehension levels. In this work, we introduce HandyLabel, a real-time annotation tool that leverages hand gesture recognition to map hand signs for labeling. The application enables users to customize gesture mappings through a web-based interface, allowing for real-time annotations. To ensure the performance of HandyLabel, we evaluate several hand gesture recognition models on an open-source hand sign (HaGRID) dataset, with and without skeleton-based preprocessing. We discovered that ResNet50 with preprocessed skeleton-based images performs an F1-score of 0.923. To validate the usability of HandyLabel, a user study was conducted with 46 participants. The results suggest that 88.9% of participants preferred HandyLabel over traditional annotation tools.

Problem

• Develops real-time annotation tool using hand gestures
• Reduces time and errors in post-processing data labeling
• Evaluates gesture recognition models for accurate real-time annotation

Innovation

• Real-time annotation tool using hand gesture recognition
• Customizable gesture mappings via web-based interface
• Skeleton-based preprocessing improves recognition model performance