🤖 AI Summary
Addressing the trade-off between performance and interpretability in summary evaluation, this paper proposes a fine-grained assessment framework based on atomic sentence decomposition: leveraging large language models (LLMs) to perform sentence-level extraction and semantic alignment between source documents and summaries, thereby constructing traceable and verifiable decision evidence chains. The method introduces the first sentence-level alignment evaluation paradigm, overcoming the limitations of conventional summary-level scoring. It is the first approach to achieve state-of-the-art (SOTA) performance while providing transparent, step-by-step reasoning paths and demonstrating robustness against hallucination. On the SummEval benchmark, it attains a consistency score of 0.580—significantly outperforming the GPT-4 evaluator (0.521)—thereby advancing both evaluation accuracy and interpretability.
📝 Abstract
Evaluating text summarization quality remains a critical challenge in Natural Language Processing. Current approaches face a trade-off between performance and interpretability. We present SEval-Ex, a framework that bridges this gap by decomposing summarization evaluation into atomic statements, enabling both high performance and explainability. SEval-Ex employs a two-stage pipeline: first extracting atomic statements from text source and summary using LLM, then a matching between generated statements. Unlike existing approaches that provide only summary-level scores, our method generates detailed evidence for its decisions through statement-level alignments. Experiments on the SummEval benchmark demonstrate that SEval-Ex achieves state-of-the-art performance with 0.580 correlation on consistency with human consistency judgments, surpassing GPT-4 based evaluators (0.521) while maintaining interpretability. Finally, our framework shows robustness against hallucination.