Investigating Temporal Motion Features for Pose-to-Text Indian Sign Language Translation

2026-09-11

AI Summary

研究通过轻量级姿态编码器和预训练T5模型，结合显式运动特征，提升印度手语到文本翻译的准确性，其中T5-small+Motion表现最佳。

Abstract

We investigate the effect of pretrained T5 model scale and explicit motion features on pose-to-text Indian Sign Language Translation (SLT) for the WSLP 2026 Shared Task. Pose sequences are projected into the embedding space of T5 through a lightweight pose encoder, with the complete model fine-tuned to generate English text. The shared task data used for this work consists of a test set with 5,334 examples and a validation set with 5,257 examples. We compare T5-small, T5-base, and T5-large, and additionally introduce a motion-augmented variant, T5-small + Motion, that adds explicit frame-to-frame pose differences to the input representation. T5-small achieves the best BLEU and ROUGE scores among the spatial-only models, while T5-large obtains the highest chrF score. Augmenting T5-small with motion features yields the largest single improvement observed in our study, substantially improving BLEU over the spatial-only baseline and making it the strongest model overall on this metric. Our submitted system ranked 5th on the official WSLP 2026 SLT testing leaderboard. The source code and trained models are publicly available on GitHub and HuggingFace.

Problem

• Pose-to-Text
• Indian Sign Language Translation
• T5 model
• Motion Features

Innovation

• T5-small + Motion
• explicit motion features
• pose-to-text translation