SignLLM: Sign Language Production Large Language Models

2024-05-17

AI Summary

This work addresses the multilingual sign language production (SLP) task by proposing SignLLM, the first end-to-end multilingual sign language large model. To overcome generalization and quality bottlenecks in cross-lingual gesture generation, SignLLM introduces a dual-generation paradigm: MLS-F (multilingual sequence-to-frame mapping) and Prompt2LangGloss (question-answering prompts to linguistic gloss mapping). It further incorporates a reinforcement learning module guided by prioritized learning channels to improve sampling efficiency of high-fidelity pose sequences. Additionally, the authors construct Prompt2Sign—the first multilingual SLP dataset supporting eight sign languages. Experiments demonstrate that SignLLM achieves state-of-the-art performance across all eight sign language benchmarks, significantly outperforming existing methods. This work is the first to empirically validate the feasibility and strong cross-lingual generalization capability of large language model–driven sign language production.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Language and Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

In this paper, we propose SignLLM, a multilingual Sign Language Production (SLP) large language model, which includes two novel multilingual SLP modes MLSF and Prompt2LangGloss that allow sign language gestures generation from query texts input and question-style prompts input respectively. Both modes can use a new RL loss based on reinforcement learning and a new RL module named Priority Learning Channel. These RL components can accelerate the training by enhancing the model's capability to sample high-quality data. To train SignLLM, we introduce Prompt2Sign, a comprehensive multilingual sign language dataset, which builds from public data, including American Sign Language (ASL) and seven others. This dataset standardizes information by extracting pose information from sign language videos into a unified compressed format. We extensively evaluate SignLLM, demonstrating that our model achieves state-of-the-art performance on SLP tasks across eight sign languages.

Problem

• Generating sign language gestures from text inputs
• Accelerating training with RL-based loss and module
• Standardizing multilingual sign language dataset for training

Innovation

• Multilingual SLP modes MLSF and Prompt2LangGloss
• RL loss and Priority Learning Channel module
• Prompt2Sign dataset with unified compressed format