Evaluation Under Imperfect Benchmarks and Ratings: A Case Study in Text Simplification

2025-04-13

AI Summary

Text simplification evaluation faces two critical bottlenecks: outdated and heterogeneous benchmark datasets that fail to reflect contemporary LLM capabilities, and low inter-annotator agreement in human evaluation, resulting in unreliable assessments and poor model discrimination. To address these, we propose SynthSimpliEval—the first high-quality synthetic benchmark tailored for the LLM era—and introduce “LLMs-as-a-jury”, a novel automated scoring paradigm wherein multiple LLMs collaboratively evaluate simplifications to enhance score stability and interpretability. Furthermore, we fine-tune learnable metrics on synthetic data, substantially improving their correlation with both human and LLM judgments. Experiments demonstrate that SynthSimpliEval significantly boosts human rater consistency and model discriminability; LLMs-as-a-jury yields robust, reproducible scores; and fine-tuned metrics approach the performance of pure LLM-based evaluation, effectively bridging the gap between learnable metrics and LLM assessment.

Technology Category

• Machine Learning: Evaluation and Analysis
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Search and Optimization: Evaluation and Analysis

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Despite the successes of language models, their evaluation remains a daunting challenge for new and existing tasks. We consider the task of text simplification, commonly used to improve information accessibility, where evaluation faces two major challenges. First, the data in existing benchmarks might not reflect the capabilities of current language models on the task, often containing disfluent, incoherent, or simplistic examples. Second, existing human ratings associated with the benchmarks often contain a high degree of disagreement, resulting in inconsistent ratings; nevertheless, existing metrics still have to show higher correlations with these imperfect ratings. As a result, evaluation for the task is not reliable and does not reflect expected trends (e.g., more powerful models being assigned higher scores). We address these challenges for the task of text simplification through three contributions. First, we introduce SynthSimpliEval, a synthetic benchmark for text simplification featuring simplified sentences generated by models of varying sizes. Through a pilot study, we show that human ratings on our benchmark exhibit high inter-annotator agreement and reflect the expected trend: larger models produce higher-quality simplifications. Second, we show that auto-evaluation with a panel of LLM judges (LLMs-as-a-jury) often suffices to obtain consistent ratings for the evaluation of text simplification. Third, we demonstrate that existing learnable metrics for text simplification benefit from training on our LLMs-as-a-jury-rated synthetic data, closing the gap with pure LLMs-as-a-jury for evaluation. Overall, through our case study on text simplification, we show that a reliable evaluation requires higher quality test data, which could be obtained through synthetic data and LLMs-as-a-jury ratings.

Problem

• Evaluating text simplification models with unreliable benchmarks.
• Addressing inconsistent human ratings in simplification evaluation.
• Improving metrics using synthetic data and LLM judges.

Innovation

• Synthetic benchmark SynthSimpliEval for reliable evaluation
• LLM judges panel for consistent auto-evaluation ratings
• Training metrics on LLM-rated synthetic data improves accuracy