🤖 AI Summary
To address two key bottlenecks in isolated sign language recognition (ISLR)—incomplete vocabulary coverage and single-view hand occlusion—this work introduces NationalCSL-DP, the first large-scale bilingual (front + left) dataset covering the entire lexicon of China’s national sign language. We propose a hybrid CNN-Transformer architecture coupled with a lightweight, highly efficient dual-view fusion strategy, incorporating spatiotemporal feature alignment, early/late fusion, and concatenation mechanisms to overcome the limitations of sequential models in capturing cross-view complementary information. Experiments on NationalCSL-DP demonstrate that our method significantly outperforms single-view baselines and various seq2seq-based fusion approaches. The results validate both the effectiveness and generalizability of dual-view data acquisition and our minimalist fusion design, establishing a new benchmark for ISLR research.
📝 Abstract
Due to the emergence of many sign language datasets, isolated sign language recognition (ISLR) has made significant progress in recent years. In addition, the development of various advanced deep neural networks is another reason for this breakthrough. However, challenges remain in applying the technique in the real world. First, existing sign language datasets do not cover the whole sign vocabulary. Second, most of the sign language datasets provide only single view RGB videos, which makes it difficult to handle hand occlusions when performing ISLR. To fill this gap, this paper presents a dual-view sign language dataset for ISLR named NationalCSL-DP, which fully covers the Chinese national sign language vocabulary. The dataset consists of 134140 sign videos recorded by ten signers with respect to two vertical views, namely, the front side and the left side. Furthermore, a CNN transformer network is also proposed as a strong baseline and an extremely simple but effective fusion strategy for prediction. Extensive experiments were conducted to prove the effectiveness of the datasets as well as the baseline. The results show that the proposed fusion strategy can significantly increase the performance of the ISLR, but it is not easy for the sequence-to-sequence model, regardless of whether the early-fusion or late-fusion strategy is applied, to learn the complementary features from the sign videos of two vertical views.