PEAR: Pairwise Evaluation for Automatic Relative Scoring in Machine Translation

2026-01-25

AI Summary

This work addresses the challenge of accurately discriminating relative translation quality in reference-free machine translation evaluation by proposing a novel pairwise comparison-based quality estimation (QE) paradigm. The approach reformulates the QE task as predicting both the direction and magnitude of quality differences between two candidate translations, using human rating deltas as supervision signals. To enhance discriminability and reduce redundancy, a sign regularization term is introduced under candidate order reversal. Employing a lightweight neural architecture and a pairwise training objective, the model outperforms larger single-candidate QE models and reference-based metrics on the WMT24 benchmark. Furthermore, it enables efficient minimum Bayes-risk (MBR) decoding with substantially reduced scoring cost while maintaining near-optimal performance.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Relational Probabilistic Models
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

We present PEAR (Pairwise Evaluation for Automatic Relative Scoring), a supervised Quality Estimation (QE) metric family that reframes reference-free Machine Translation (MT) evaluation as a graded pairwise comparison. Given a source segment and two candidate translations, PEAR predicts the direction and magnitude of their quality difference. The metrics are trained using pairwise supervision derived from differences in human judgments, with an additional regularization term that encourages sign inversion under candidate order reversal. On the WMT24 meta-evaluation benchmark, PEAR outperforms strictly matched single-candidate QE baselines trained with the same data and backbones, isolating the benefit of the proposed pairwise formulation. Despite using substantially fewer parameters than recent large metrics, PEAR surpasses far larger QE models and reference-based metrics. Our analysis further indicates that PEAR yields a less redundant evaluation signal relative to other top metrics. Finally, we show that PEAR is an effective utility function for Minimum Bayes Risk (MBR) decoding, reducing pairwise scoring cost at negligible impact.

Problem

• Machine Translation
• Quality Estimation
• Pairwise Evaluation
• Reference-free Evaluation
• Relative Scoring

Innovation

• Pairwise Evaluation
• Quality Estimation
• Machine Translation
• Relative Scoring
• Minimum Bayes Risk