Cross-Sign Language Transfer Learning Using Domain Adaptation with Multi-scale Temporal Alignment

2026-08-17

AI Summary

This study addresses the challenges of data scarcity and cross-lingual transfer in sign language recognition by proposing a multi-scale temporal relation alignment mechanism based on Temporal Relation Networks (TRN). Integrating TA3N domain adaptation with optical flow features, this approach effectively mitigates domain shift through multi-scale temporal modeling, thereby significantly enhancing RGB modality representations. Experimental results demonstrate that the proposed framework outperforms conventional transfer learning methods in American Sign Language (ASL) recognition tasks. These findings validate the efficacy of RGB-based approaches in low-resource scenarios and provide an efficient solution for cross-lingual sign language recognition, offering a robust pathway to overcome resource limitations in the field.

Abstract

Sign language serves as a vital means of communication for individuals with hearing impairments, yet recognition resources for the over 100 distinct sign languages are severely lacking. In response, we present our work on sign language recognition using transfer learning and the domain adaptation method TA3N, which utilizes the Temporal Relational Network (TRN) module for aligning multi-scale temporal relations. Our findings highlight the superior performance of Domain Adaptation to neural network-based transfer learning, particularly in improving recognition of American Sign Language (ASL). Our research also identifies the effectiveness of aligning shorter-term temporal features between source and target domains. In addition to using RGB, we conducted experiments using Optical Flow mode for the sign language samples, ultimately determining that RGB outperforms Optical Flow in the majority of cases. Our work aims to improve accessibility and communication for individuals who rely on sign language as their primary mode of communication.

Problem

• Sign Language Recognition
• Cross-Sign Language Transfer Learning
• Domain Adaptation
• Resource Scarcity

Innovation

• Domain Adaptation
• Multi-scale Temporal Alignment
• Temporal Relational Network
• Cross-Sign Language Transfer Learning
• Sign Language Recognition