DiscoSign: Discourse-Aware Text to Sign Language Gloss Translation

2026-09-02

AI Summary

本文提出DiscoSign，一种基于大语言模型的文本到手语词汇翻译方法，解决了空间共指、问答从句和概念-词汇一致性等话语现象问题。

Abstract

Sign language processing systems have traditionally operated at the sentence level, ignoring critical discourse phenomena fundamental to sign language comprehension. We introduce DiscoSign, a computational approach for discourse-aware text to sign language gloss translation grounded in linguistic research. We address three key phenomena within our modular Large Language Model (LLM)-based translation framework: (i) spatial coreference resolution, where entities maintain consistent spatial locations throughout discourse; (ii) Question-Answer Clauses (QACs), pseudocleft structures serving specific discourse functions; and (iii) concept-gloss consistency, ensuring stable mappings between English concepts and American Sign Language (ASL) signs. Traditional translation metrics fail to capture discourse-level quality, so we introduce a suite of novel evaluation metrics designed to assess each dimension of discourse coherence addressed by our framework. Experiments on sentence-level and discourse-level datasets show that our approach for discourse-aware processing significantly improves spatial consistency and entity tracking relative to sentence-only translation, while maintaining competitive single-sentence gloss translation quality. Our work establishes the first systematic framework for discourse-level text to sign language gloss translation with corresponding evaluation methodology.

Problem

• discourse phenomena
• sign language gloss translation
• spatial coreference resolution
• Question-Answer Clauses (QACs)
• concept-gloss consistency

Innovation

• discourse-aware
• spatial coreference resolution
• Question-Answer Clauses
• concept-gloss consistency