DoCIA: An Online Document-Level Context Incorporation Agent for Speech Translation

2025-04-07

AI Summary

To address discourse incoherence in speech translation (ST) caused by ASR noise, this paper proposes an online document-level context fusion framework. Methodologically, it introduces a lightweight, multi-level document context encoder with dynamic context injection across three stages: ASR refinement, translation, and post-editing; additionally, it incorporates an LLM-assisted module featuring a novel hallucination-averse adaptive post-editing decision strategy. The key contributions are: (1) the first low-overhead, highly robust online document context modeling approach for ST; and (2) consistent improvements over strong baselines across four mainstream LLMs, achieving significant gains in both sentence-level BLEU and document-level coherence metrics—demonstrating the efficacy of document-level context in enhancing noisy speech translation.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Document-level context is crucial for handling discourse challenges in text-to-text document-level machine translation (MT). Despite the increased discourse challenges introduced by noise from automatic speech recognition (ASR), the integration of document-level context in speech translation (ST) remains insufficiently explored. In this paper, we develop DoCIA, an online framework that enhances ST performance by incorporating document-level context. DoCIA decomposes the ST pipeline into four stages. Document-level context is integrated into the ASR refinement, MT, and MT refinement stages through auxiliary LLM (large language model)-based modules. Furthermore, DoCIA leverages document-level information in a multi-level manner while minimizing computational overhead. Additionally, a simple yet effective determination mechanism is introduced to prevent hallucinations from excessive refinement, ensuring the reliability of the final results. Experimental results show that DoCIA significantly outperforms traditional ST baselines in both sentence and discourse metrics across four LLMs, demonstrating its effectiveness in improving ST performance.

Problem

• Incorporates document-level context in speech translation
• Addresses noise from automatic speech recognition in translation
• Prevents hallucinations from excessive refinement in translation

Innovation

• Online framework integrating document-level context
• Multi-level document info with low overhead
• Determination mechanism prevents excessive refinement