Sign Language Recognition Using Original and Synthetic Depth Image Based Point Cloud Data Models

2026-08-10

AI Summary

This study addresses the limitation in existing sign language recognition research, which predominantly relies on RGB images and lacks access to real depth data, thereby hindering the application of point cloud–based methods. For the first time, it systematically compares the performance of point clouds derived from real versus synthetic depth maps for sign language recognition. Specifically, synthetic depth maps are generated from RGB images using Depth Anything V2, and point clouds are constructed from both synthetic and real depth data. These point clouds are then evaluated using representative models including PointNet variants, LSTM, and Point Gesture Map. Experimental results demonstrate that point clouds from synthetic depth consistently achieve recognition accuracy on par with or even surpassing those from real depth across multiple architectures, confirming their feasibility and practical utility as a viable alternative in scenarios where genuine depth sensors are unavailable.

Abstract

Research regarding the sign language recognition mostly relies on RGB images, whileas sign language datasets that provide depth images are limited. Point clouds obtained from depth images can be used for sign language recognition with neural networks like PointNet. In recent years, various neural networks are used for generating realistic depth images from monocular RGB images. In this work, synthetic depth images were created from RGB images using Depth Anything V2 network. For this purpose, three sign language datasets (Real-time ASL Fingerspelling, KArSL, AUTSL) which contain both RGB and depth images were used. Classification accuracies of the point cloud data created from both original and synthetic depth images using various PointNet architectures were measured for sign language recognition. From the original and synthetic point clouds, frame based, Point Gesture Map and Long Short Term Memory data models were used for classification and their performances were compared. In the results, both original and synthetic based data achieved acceptable performance in most models. In general, original depth based point cloud models performed better than synthetic ones, however in some models synthetic depth based models performed better than the originals.

Problem

• Sign Language Recognition
• Depth Image
• Point Cloud
• Synthetic Data
• RGB-to-Depth Translation

Innovation

• synthetic depth images
• point cloud
• sign language recognition
• Depth Anything V2
• PointNet