The Impact of VAE Design on Latent Pose Representations for Diffusion-based Sign Language Production

2026-06-22

AI Summary

This study addresses a critical gap in sign language generation research, where variational autoencoders (VAEs) are typically evaluated solely on reconstruction quality while overlooking how their latent space structure influences downstream diffusion models. The work systematically investigates how VAE architectures and training objectives shape latent representations of sign language poses, demonstrating that structural properties of the latent space—rather than reconstruction fidelity—are more predictive of performance in text-to-sign generation. By integrating geometric metrics with post-hoc BLEU scores on the Phoenix14T dataset, the authors show that different VAE designs significantly affect generation quality, thereby highlighting the pivotal role of latent space geometry. These findings offer a novel perspective on representation learning for sign language generation, emphasizing the need to prioritize latent structure over pixel-level reconstruction accuracy.

Abstract

Latent diffusion approaches to sign language production (SLP) rely on an initial stage that learns an encoding of sign pose sequences, enabling generative modeling in the resulting latent space. The autoencoder used in this stage is typically evaluated in terms of reconstruction quality using geometric metrics common in SLP. While informative, these metrics do not fully capture latent space properties that may influence the training and performance of the downstream generative model. In this work, we investigate how architectural and training objective design choices in a variational autoencoder (VAE) for sign pose encoding affect latent space structure, and how these differences translate into the performance of a latent diffusion model for text-to-sign generation. Our experiments on Phoenix14T dataset show that variations in generative performance, measured through back-translation BLEU scores, can sometimes be better explained by differences in latent space properties than by VAE reconstruction accuracy alone.

Problem

• variational autoencoder
• latent space
• sign language production
• diffusion model
• pose representation

Innovation

• variational autoencoder
• latent space structure
• diffusion-based sign language production
• pose representation
• generative modeling