Sign Language Video Synthesis via Loss-Guided Multi-Expert GANs

2026-08-13

AI Summary

This work addresses the challenges of insufficient detail and low visual quality in sign language video generation by proposing a multi-expert GAN architecture. The approach employs three specialized discriminators—global, hand, and head—to collaboratively guide the generator, complemented by a dual-path convolution–Transformer structure and a learnable feature fusion module that enables implicit regional specialization. To stabilize training with multiple discriminators, a United Loss consensus mechanism is introduced alongside an alternating three-stage training strategy. Evaluated on a self-collected 156 GB dataset, the proposed models with 0.2B and 1.3B parameters achieve PSNR scores of 29.8 and 30.7, respectively, while requiring only 1.5 GB of inference GPU memory, thereby enabling deployment on consumer-grade hardware.

Abstract

This preliminary technical report presents a framework for sign language video synthesis using a loss-guided multi-expert Generative Adversarial Network (GAN) to enhance communication for individuals with hearing impairments. Three specialized discriminators -- global, hand, and head -- each guide a corresponding expert branch in the generator toward a distinct visual region, enabling implicit feature specialization without explicit diversity losses. To stabilize this multi-discriminator system, whose early-phase training otherwise exhibits chaotic dynamics, we introduce a United Loss consensus mechanism that regularizes each discriminator toward the ensemble average at a 10% weight. Each branch further adopts a dual-pathway convolutional-transformer design with learnable AdaptiveFeatureFusion, balancing the stability of convolutions against the detail of windowed self-attention. The generator is trained using an alternating three-mode schedule (discriminator, holistic generation, branch-specialized generation). On a custom 156GB dataset with a filtered test set that removes easy and repetitive samples, our 0.2B-parameter variant achieves 29.8 PSNR and the 1.3B-parameter variant achieves 30.7 PSNR, with inference VRAM footprints of 1.5 GB and 8 GB respectively, enabling deployment on consumer-grade hardware. Full ablation studies remain ongoing due to the 2-3 month training cycle on a single GPU. The system was showcased at the 2025 Hong Kong Frontier Technology Summit.

Problem

• Sign Language Video Synthesis
• Hearing Impairment
• Video Generation
• Communication Accessibility

Innovation

• Multi-Expert GAN
• Loss-Guided Training
• AdaptiveFeatureFusion
• Dual-Pathway Convolutional-Transformer
• United Loss Consensus