Hermes the Polyglot: A Unified Framework to Enhance Expressiveness for Multimodal Interlingual Subtitling

2026-01-31

AI Summary

This work proposes the first context-aware large language model (LLM) framework for cross-lingual subtitle translation that integrates speaker diarization, terminology recognition, and expressive enhancement to address challenges such as weak semantic coherence, inaccurate pronoun and term translation, and unnatural phrasing. By jointly modeling dialogue context and pragmatic features through a multi-module architecture, the approach significantly improves the coherence, accuracy, and fluency of translated subtitles. Experimental results demonstrate that the system achieves state-of-the-art performance in speaker diarization and generates semantically consistent and naturally expressed cross-lingual subtitles, establishing a new paradigm for multimodal subtitle translation.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Language and Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Interlingual subtitling, which translates subtitles of visual media into a target language, is essential for entertainment localization but has not yet been explored in machine translation. Although Large Language Models (LLMs) have significantly advanced the general capabilities of machine translation, the distinctive characteristics of subtitle texts pose persistent challenges in interlingual subtitling, particularly regarding semantic coherence, pronoun and terminology translation, and translation expressiveness. To address these issues, we present Hermes, an LLM-based automated subtitling framework. Hermes integrates three modules: Speaker Diarization, Terminology Identification, and Expressiveness Enhancement, which effectively tackle the above challenges. Experiments demonstrate that Hermes achieves state-of-the-art diarization performance and generates expressive, contextually coherent translations, thereby advancing research in interlingual subtitling.

Problem

• interlingual subtitling
• semantic coherence
• pronoun translation
• terminology translation
• translation expressiveness

Innovation

• Interlingual Subtitling
• Large Language Models
• Speaker Diarization
• Terminology Identification
• Expressiveness Enhancement