AllSummedUp: un framework open-source pour comparer les metriques d'evaluation de resume

2025-08-29

AI Summary

This paper addresses core challenges in automatic text summarization evaluation—poor metric reproducibility, low correlation with human judgments, and the trade-off between computational cost and result stability. To this end, we introduce the first open-source, unified evaluation framework supporting standardized, reproducible comparison of diverse metrics, including ROUGE, G-Eval, SEval-Ex, and multiple LLM-based evaluators. Systematic experiments on SummEval reveal that LLM-based metrics exhibit substantial stochasticity and irreproducibility; quantitative analysis further confirms that high human alignment often comes at the cost of elevated computational overhead and reduced output stability. Our key contributions are: (1) the first unified framework compatible with heterogeneous evaluation paradigms; (2) the first empirical demonstration of reproducibility deficits in LLM-based evaluation; and (3) advancement of transparent, standardized evaluation protocols, establishing a reproducible benchmark for future summarization evaluation research.

Technology Category

• Natural Language Processing: Summarization
• Machine Learning: Evaluation and Analysis
• Search and Optimization: Evaluation and Analysis

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success

Abstract

This paper investigates reproducibility challenges in automatic text summarization evaluation. Based on experiments conducted across six representative metrics ranging from classical approaches like ROUGE to recent LLM-based methods (G-Eval, SEval-Ex), we highlight significant discrepancies between reported performances in the literature and those observed in our experimental setting. We introduce a unified, open-source framework, applied to the SummEval dataset and designed to support fair and transparent comparison of evaluation metrics. Our results reveal a structural trade-off: metrics with the highest alignment with human judgments tend to be computationally intensive and less stable across runs. Beyond comparative analysis, this study highlights key concerns about relying on LLMs for evaluation, stressing their randomness, technical dependencies, and limited reproducibility. We advocate for more robust evaluation protocols including exhaustive documentation and methodological standardization to ensure greater reliability in automatic summarization assessment.

Problem

• Investigating reproducibility challenges in automatic text summarization evaluation
• Highlighting discrepancies between reported and observed metric performances
• Addressing randomness and limited reproducibility in LLM-based evaluation methods

Innovation

• Open-source framework for metric comparison
• Applied to SummEval dataset analysis
• Highlights LLM evaluation limitations and trade-offs