Rubric-as-Experts: Case-Specific MQM Rubrics for Translation Quality Evaluation

2026-06-19

AI Summary

Existing static, shared MQM scoring rules struggle to accommodate variations across translation samples in error complexity, ambiguity, and required evaluation granularity, thereby limiting fine-grained error detection performance. This work proposes a dynamic scoring framework that, for the first time, preserves the MQM’s predefined error-type taxonomy while adaptively constructing an evaluation subspace for each translation instance to dynamically select the optimal error subtypes and granularity levels. By integrating large language models for span-level error localization and quality estimation, the approach achieves significant improvements in the Matthews Correlation Coefficient (MCC) across multiple WMT span-level quality estimation benchmarks, yielding more accurate and cleaner error identification compared to fixed-rule baselines.

Abstract

Large language models (LLMs) have shown strong potential in fine-grained translation quality evaluation (QE), yet existing MQM-based approaches typically rely on fixed rubric configurations shared across all translation samples. However, translation instances often differ substantially in error complexity, ambiguity, and required evaluation granularity, making static rubric allocation suboptimal for span-level error detection. We find that larger MQM subtype spaces improve error coverage but also introduce more false positives, while different translation instances prefer different rubric granularities, suggesting that evaluation spaces should be allocated dynamically for each case. Motivated by these observations, we propose a case-specific dynamic rubric framework that adaptively constructs MQM evaluation spaces for individual translation instances. Unlike fully free-form rubric generation methods, our framework remains grounded in the predefined MQM taxonomy while dynamically selecting suitable subtype spaces and evaluation granularity for different cases. Experiments on WMT span-level QE benchmarks across multiple model scales demonstrate that the proposed framework consistently improves MCC and produces cleaner span-level error localization compared with static rubric settings. Our results suggest that combining structured MQM rubrics with case-specific adaptive allocation is an effective strategy for fine-grained LLM-based translation evaluation.

Problem

• translation quality evaluation
• MQM rubrics
• case-specific
• span-level error detection
• dynamic rubric allocation

Innovation

• dynamic rubric
• case-specific evaluation
• MQM
• translation quality evaluation
• LLM-based QE