Gloss-Free Representation Learning for Cross-Dataset Sign Spotting

2026-08-11

AI Summary

This work addresses key challenges in resource-constrained sign language research—namely, the absence of dense lexical annotations, loose alignment between spoken and signed content in broadcast news, and fragmented pseudo-labels caused by the morphological complexity of languages like Turkish. To overcome these issues, the authors propose a weakly supervised pretraining approach that requires no manual gloss annotations. By integrating rule-based lemmatization with large language model (LLM)-constrained lexical normalization, they generate high-quality pseudo-gloss labels and pretrain a transferable sign language encoder on a newly curated TSL-News corpus. This study presents the first application of transcription-based, gloss-free pretraining to cross-dataset sign language localization, achieving substantial performance gains: the top-5 mean IoU improves from 0.235 to 0.465 (with 56.2% of samples attaining IoU ≥ 0.5), while downstream translation scores reach 11.04 BLEU-4 and 27.43 ROUGE.

Abstract

Sign-language research for resource-constrained languages is often limited by the cost of dense linguistic labels such as glosses, temporal boundaries, and sign order. Broadcast news offers a practical alternative by pairing continuous signing with spoken-language transcripts, but this supervision is weak since text and signing are loosely aligned. Morphologically rich languages such as Turkish add further difficulty, as the same lexical meaning can appear in many inflected forms while some derived forms should remain distinct. We study whether weak transcript-based supervision can pretrain a reusable sign encoder in this setting, where poor text normalization can fragment pseudo-gloss targets and weaken representation learning. Unlike prior pseudo-gloss pipelines designed mainly to improve translation, we test whether the pretrained encoder transfers as a reusable representation for cross-dataset sign spotting. We pretrain on TSL-News, a new Turkish broadcast corpus, using pseudo-gloss labels derived from transcripts rather than manual annotation, comparing rule-based morphological lemmatization with constrained LLM-assisted normalization over a fixed vocabulary. We evaluate the learned representations via cross-dataset sign spotting on a new TSL Spotting Benchmark built from the TSL Dictionary corpus. The LLM-assisted encoder raises top-5 temporal localization mean IoU from 0.235 to 0.465, with 56.2% of examples reaching an IoU of at least 0.50; a frequency analysis suggests this gain is not mainly driven by memorizing frequent pseudo-gloss labels. In a downstream translation check, the same pretraining improves BLEU-4 from 9.60 to 11.04 and ROUGE from 23.48 to 27.43. These results show that loosely aligned broadcast data can provide effective weak supervision for learning sign representations that capture both lexical content and temporal structure.

Problem

• sign language
• weak supervision
• cross-dataset
• morphologically rich languages
• sign spotting

Innovation

• weak supervision
• pseudo-gloss
• sign spotting
• morphological normalization
• cross-dataset transfer