SignNet-1M: Large-Scale Multilingual Sign Language Video Dataset with Downstream Benchmarks

2026-06-23

AI Summary

Existing sign language recognition models suffer from limited generalization due to training data constrained to a single viewpoint, background, and signer identity. To address this, this work introduces SignNet-1M, the first large-scale multilingual sign language video dataset, and proposes a novel three-axis augmentation strategy that systematically integrates 3D generation (via 3D Gaussian Splatting), identity and background editing, and video-level photorealistic perturbations. Leveraging diffusion models, the approach synthesizes high-fidelity, highly diverse pseudo-signing videos to establish a unified multitask evaluation benchmark. Experiments demonstrate that the proposed method substantially enhances model robustness under out-of-distribution scenarios—including cross-viewpoint, cross-background, cross-identity, and compression-induced perturbations—while preserving performance on in-distribution data.

Abstract

Sign language models are typically trained on datasets captured under constrained conditions, with limited viewpoint, background, and signer-identity diversity, leading to poor robustness under real-world distribution shifts. We introduce SignNet-1M, a large-scale augmented dataset spanning ASL, CSL, and German Sign Language (DGS). SignNet-1M synthesizes realistic variations along three axes: (i) novel-view rendering (rotation and zoom) via 3D Gaussian Splatting (3DGS), (ii) scene/identity editing via diffusion models for background replacement and signer substitution while preserving sign motion and linguistic content, and (iii) post-rendering augmentations that emulate capture and compression artifacts (e.g., pose/temporal perturbations and video-level corruptions) to better match in-the-wild recordings. Beyond data release, we provide a unified benchmark suite across downstream tasks (e.g., translation and recognition) and ablations that isolate each augmentation component. Experiments across backbones show that training with SignNet-1M consistently improves generalization under cross-view, cross-background, cross-identity, and post-rendering shifts, while maintaining strong in-distribution performance. The dataset, full augmentation pipeline, and benchmark are available at https://signnet.chatsign.ai/.

Problem

• sign language
• dataset diversity
• distribution shift
• real-world robustness
• multilingual

Innovation

• 3D Gaussian Splatting
• diffusion-based editing
• multilingual sign language dataset
• realistic data augmentation
• distribution shift robustness