IndicQE-APE: A Benchmark for Quality Estimation and Automatic Post-Editing for Indic Languages

2026-08-17

AI Summary

This study addresses the fragmentation of quality estimation and automatic post-editing data for Indian languages by constructing a unified multi-label benchmark comprising 126,000 instances. Through multi-source data integration and stratified sampling, we systematically evaluate large language models and COMET metrics. Results reveal that conflicts between sentence-level and token-level signals serve as a reliable difficulty dimension, while few-shot prompting induces performance degradation and optimal monolingual metrics fail to generalize across language pairs. By establishing standardized evaluation protocols for low-resource translation quality research, this work provides critical empirical evidence to guide future model optimization and metric development.

Abstract

Indic quality estimation (QE) and automatic post-editing (APE) data is spread across separate releases, so no single resource supports training and evaluation across tasks and language pairs on one footing. We consolidate the WMT 2020--2024 shared-task lineage with an extended English--Malayalam resource into \indicqe: $126{,}754$ instances over nine directional pairs, with up to four label types aligned on the same segment, a direct assessment, a human post-edit, word-level OK/BAD tags and an error explanation, and a test set stratified over four difficulty axes. On it, we benchmark six prompted LLMs and three COMET metrics on segment-level QE, and three systems on APE. Two of the axes are defined partly on the direct assessment and select a compressed slice of it, so each axis is compared against a control drawn from the same language pair with the same score distribution. Only one survives that control: segments whose holistic and token-level quality signals conflict are ranked worse than equally-scored segments of the same language, for all nine systems and all seven pairs that carry the axis. Annotator disagreement, which looks second-hardest without the control, has no effect with it. Few-shot prompting costs every model $\leq$ $3.4$B both correlation and output-format compliance. Within-language accuracy does not make scores comparable across pairs: of the three trained metrics, the one with the best within-language correlation loses most when the pairs are pooled. The benchmark and code will be released.

Problem

• Indic languages
• Quality Estimation
• Automatic Post-Editing
• Benchmark
• Data consolidation

Innovation

• IndicQE Benchmark
• Quality Estimation
• Automatic Post-Editing
• Multi-label Alignment
• Difficulty-stratified Evaluation