Target-Side Paraphrase Augmentation for Sign Language Translation with Large Language Models

2026-05-29

AI Summary

This study addresses the challenges of sign language translation, particularly the scarcity of parallel data and the long-tailed distribution of target vocabulary. The authors propose a novel approach that leverages the large language model GPT-4o to generate semantically preserved, target-controlled paraphrases, integrated with the Signformer pose-encoding architecture through a two-stage training strategy: pretraining on augmented corpora followed by fine-tuning on original data. To better evaluate semantic fidelity, they introduce an LLM-as-a-Judge mechanism, which reveals that conventional word-overlap metrics substantially underestimate actual improvements in translation quality. Experimental results demonstrate that the method achieves a BLEU-4 score of 10.33 (+0.77) on PHOENIX14T and exhibits strong generalization across GSL and LSA-T datasets, significantly enhancing semantic faithfulness.

Abstract

Sign language translation (SLT) remains constrained by limited paired sign-video/text corpora and heavy-tailed target vocabularies. We study target-side augmentation in which GPT-4o generates controlled paraphrase variants of reference sentences while the sign input remains unchanged. A Signformer-style pose-based Transformer is trained under a two-stage schedule: pre-training on the augmented corpus followed by fine-tuning on the original references. We evaluate on three datasets spanning complementary challenges: PHOENIX14T (German Sign Language), with moderate lexical diversity; GSL (Greek Sign Language), with highly ontrolled, repetitive recordings; and LSA-T (Argentinian Sign Language), with severe long-tail sparsity. On PHOENIX14T, augmentation improves BLEU-4 from 9.56 to 10.33. The near-saturated GSL baseline and extremely sparse LSA-T setting reveal the limits of the approach. To our knowledge, this is the first study to apply LLM-generated target-side araphrases and LLM-as-a-Judge evaluation to SLT. The semantic evaluation reveals gains in fidelity that lexical overlap metrics understate.

Problem

• sign language translation
• limited paired corpora
• long-tailed vocabulary
• target-side augmentation

Innovation

• target-side paraphrase augmentation
• large language models
• sign language translation
• LLM-as-a-Judge
• pose-based Transformer