Physics-Informed Diffusion for Biomechanically Plausible 3D Sign Language Generation

2026-07-16

AI Summary

This work addresses the common imbalance in existing 3D sign language generation methods between semantic fidelity and biomechanical plausibility, which often results in unnatural motions such as skeletal drift, joint violations, and rigid finger movements. To resolve this, we propose PIDiffSign—a physics-informed diffusion model incorporating anatomical constraints—built upon a Transformer encoder-decoder architecture that embeds biomechanical priors directly into both the diffusion process and training objective, thereby enabling the first joint optimization of semantic alignment and kinematic realism. Leveraging timestep-adaptive normalization, a differentiable geometric module, a contrastive lexical-pose alignment loss, and classifier-free guidance sampling, our method significantly outperforms current diffusion-based baselines on the PHOENIX14T and CSL-Daily benchmarks, achieving consistent improvements in pose accuracy, joint angle correctness, motion distribution fidelity, and back-translation quality.

Abstract

Sign language production, which generates continuous 3D skeletal motion from spoken language input, must simultaneously satisfy two constraints: semantic fidelity, so that a deaf viewer can recognize the intended sequence of glosses, and biomechanical plausibility, so that the generated skeleton respects anatomical constraints. Existing approaches optimize semantic reconstruction through coordinate-based objectives that treat the skeleton as an unstructured vector, thus allowing for bone length drift, joint angle violations, and temporarily locked fingers. We introduce PIDiffSign, a physics-informed diffusion model for gloss-to-pose translation that incorporates anatomical constraints into both the architecture and training objective. The model uses a Transformer encoder-decoder, where the decoder is conditioned on the diffusion time step through adaptive zero-initialized layer normalization and cross-attends to gloss representations. A differentiable geometry module enforces bone length consistency and biologically valid joint angles throughout generation. Training combines anthropomorphic, kinematic, angular, and finger-joint constraints with a contrastive gloss-pose alignment loss and classifier-free guidance for semantically conditioned sampling. Experiments on the PHOENIX14T and CSL-Daily benchmarks show consistent improvements over a strong diffusion baseline in pose accuracy, joint-angle correctness, distributional realism, and back-translation quality. These results demonstrate that physics-informed diffusion improves both motion realism and semantic fidelity for sign language generation.

Problem

• sign language generation
• biomechanical plausibility
• 3D skeletal motion
• anatomical constraints
• semantic fidelity

Innovation

• physics-informed diffusion
• biomechanical constraints
• 3D sign language generation
• differentiable geometry module
• gloss-to-pose translation