Do LLMs Truly Benefit from Longer Context in Automatic Post-Editing?

2026-01-27

AI Summary

This study investigates whether large language models (LLMs) genuinely benefit from document-level context in automatic post-editing (APE) tasks. Through systematic comparisons of closed-source and open-source LLMs within a unified one-shot prompting framework—augmented by both automatic metrics and human evaluation, as well as data poisoning attacks to assess robustness—the findings reveal that, despite achieving near-human performance in translation quality and robustness, closed-source models scarcely leverage document-level context to correct discourse-sensitive errors. Furthermore, their high inference costs and latency hinder practical deployment. The results underscore the critical need for efficient long-context modeling in APE and expose a significant discrepancy between automatic evaluation scores and human judgments.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Automatic post-editing (APE) aims to refine machine translations by correcting residual errors. Although recent large language models (LLMs) demonstrate strong translation capabilities, their effectiveness for APE--especially under document-level context--remains insufficiently understood. We present a systematic comparison of proprietary and open-weight LLMs under a naive document-level prompting setup, analyzing APE quality, contextual behavior, robustness, and efficiency. Our results show that proprietary LLMs achieve near human-level APE quality even with simple one-shot prompting, regardless of whether document context is provided. While these models exhibit higher robustness to data poisoning attacks than open-weight counterparts, this robustness also reveals a limitation: they largely fail to exploit document-level context for contextual error correction. Furthermore, standard automatic metrics do not reliably reflect these qualitative improvements, highlighting the continued necessity of human evaluation. Despite their strong performance, the substantial cost and latency overheads of proprietary LLMs render them impractical for real-world APE deployment. Overall, our findings elucidate both the promise and current limitations of LLM-based document-aware APE, and point toward the need for more efficient long-context modeling approaches for translation refinement.

Problem

• automatic post-editing
• large language models
• document-level context
• contextual error correction
• translation refinement

Innovation

• large language models
• automatic post-editing
• document-level context
• contextual robustness
• human evaluation