Exploiting Ensemble Learning for Cross-View Isolated Sign Language Recognition

2025-02-04

AI Summary

This paper addresses the weak generalization of front-view data in cross-view isolated sign language recognition (CV-ISLR), caused by substantial camera viewpoint variations. To tackle this challenge, we propose the first ensemble learning framework specifically designed for CV-ISLR. Methodologically, we innovatively integrate multi-view feature consistency modeling with the multi-dimensional spatiotemporal representations of Video Swin Transformer, constructing a unified RGB and RGB-D multimodal joint learning architecture. Furthermore, we introduce a dual-path ensemble strategy operating at both model-level and feature-level to significantly enhance cross-view robustness. Evaluated on the WWW 2025 CV-ISLR Challenge, our approach achieves third place in both the RGB and RGB-D tracks, demonstrating its effectiveness and practical applicability.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

In this paper, we present our solution to the Cross-View Isolated Sign Language Recognition (CV-ISLR) challenge held at WWW 2025. CV-ISLR addresses a critical issue in traditional Isolated Sign Language Recognition (ISLR), where existing datasets predominantly capture sign language videos from a frontal perspective, while real-world camera angles often vary. To accurately recognize sign language from different viewpoints, models must be capable of understanding gestures from multiple angles, making cross-view recognition challenging. To address this, we explore the advantages of ensemble learning, which enhances model robustness and generalization across diverse views. Our approach, built on a multi-dimensional Video Swin Transformer model, leverages this ensemble strategy to achieve competitive performance. Finally, our solution ranked 3rd in both the RGB-based ISLR and RGB-D-based ISLR tracks, demonstrating the effectiveness in handling the challenges of cross-view recognition. The code is available at: https://github.com/Jiafei127/CV_ISLR_WWW2025.

Problem

• Addresses cross-view isolated sign language recognition
• Enhances model robustness with ensemble learning
• Leverages multi-dimensional Video Swin Transformer

Innovation

• Ensemble learning robustness
• Multi-angle gesture understanding
• Video Swin Transformer model