Enhancing Entertainment Translation for Indian Languages using Adaptive Context, Style and LLMs

2024-12-29

AI Summary

This work addresses the insufficient modeling of dialogue style and contextual nuance in entertainment translation—particularly for Indian languages like Hindi—in tasks such as dubbing and subtitling. We propose the first adaptive, context- and style-aware framework specifically designed for entertainment translation. Our method decouples three core components: dialogue-level contextual modeling, fine-grained stylistic representation learning, and large language model (LLM) prompt engineering. It introduces a language- and model-agnostic dynamic prompt generation algorithm that supports heterogeneous input sources and generalizes across languages and LLMs. Empirical evaluation shows significant improvements over multiple state-of-the-art LLMs on the COMET metric; human evaluations further confirm consistent superiority in translation naturalness, faithfulness, and expressiveness, with higher win rates against all baselines.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

We address the challenging task of neural machine translation (NMT) in the entertainment domain, where the objective is to automatically translate a given dialogue from a source language content to a target language. This task has various applications, particularly in automatic dubbing, subtitling, and other content localization tasks, enabling source content to reach a wider audience. Traditional NMT systems typically translate individual sentences in isolation, without facilitating knowledge transfer of crucial elements such as the context and style from previously encountered sentences. In this work, we emphasize the significance of these fundamental aspects in producing pertinent and captivating translations. We demonstrate their significance through several examples and propose a novel framework for entertainment translation, which, to our knowledge, is the first of its kind. Furthermore, we introduce an algorithm to estimate the context and style of the current session and use these estimations to generate a prompt that guides a Large Language Model (LLM) to generate high-quality translations. Our method is both language and LLM-agnostic, making it a general-purpose tool. We demonstrate the effectiveness of our algorithm through various numerical studies and observe significant improvement in the COMET scores over various state-of-the-art LLMs. Moreover, our proposed method consistently outperforms baseline LLMs in terms of win-ratio.

Problem

• Indian Language
• Neural Machine Translation
• Dialogue Style Processing

Innovation

• Neural Machine Translation
• Contextual and Stylistic Awareness
• Entertainment Dialogue Translation