UniTranslator: A Unified Multi-modal Framework for End-to-end In-Image Machine Translation

2026-06-23

AI Summary

This work addresses the challenges of semantic inconsistency and spatial misalignment between translation understanding and text generation in image-based machine translation. To this end, the authors propose a unified end-to-end multimodal framework that tightly couples translation comprehension with text re-rendering. The core innovations include a Understanding-Generation Alignment Module (UGAM) to ensure semantic consistency and a Spatial Mask Decoder (SMD) with pixel-level supervision for precise localization. The proposed method achieves state-of-the-art performance across multiple benchmarks, demonstrating robustness in multilingual settings and complex real-world layouts. Furthermore, the study empirically validates the mutual enhancement between translation understanding and image generation within the proposed architecture.

Abstract

In-Image Machine Translation (IIMT) aims to translate scene text in an image and render the translated text back into the original regions while preserving the overall visual appearance. Recent unified multimodal models provide a promising solution by combining visual-text understanding and image generation within a single framework. However, directly adapting such models to IIMT remains challenging. In particular, they often suffer from understanding-generation conflicts, where the translation inferred during understanding is inconsistent with the text supervision used in generation, and spatial position misalignment, where the rendered text does not accurately match the target text regions. To address these issues, we present UniTranslator, a unified multimodal framework for IIMT that tightly couples translation understanding and text editing. Specifically, we introduce an Understand-Generation Alignment Module (UGAM) to bridge the representation gap between understanding and generation, encouraging semantic consistency between translated content prediction and text rendering. We further propose a Spatial Mask Decoder (SMD) with pixel-level supervision over text regions to improve spatial grounding, geometric alignment, and layout controllability during generation. Extensive experiments on multiple benchmarks demonstrate that UniTranslator achieves state-of-the-art performance across diverse language directions and complex real-world layouts. Moreover, our results reveal a strong mutual reinforcement effect between translation understanding and image generation, highlighting the advantage of unified translation multimodal learning. Code is available at https://github.com/SeerRay-Lab/Unitranslator.

Problem

• In-Image Machine Translation
• understanding-generation conflict
• spatial position misalignment
• scene text translation
• visual appearance preservation

Innovation

• Understand-Generation Alignment
• Spatial Mask Decoder
• In-Image Machine Translation
• Unified Multimodal Framework
• Pixel-level Supervision