🤖 AI Summary
This work addresses the challenge that existing automatic evaluation metrics struggle to accurately assess the factual consistency of opinion summaries generated by large language models. To this end, the authors propose FactSim, an end-to-end fully automated evaluation method that extracts factual claims from both the generated summary and the original user reviews, and introduces a robust fact similarity scoring mechanism designed to handle negations, paraphrases, and elaborations. This approach effectively measures both factual consistency and coverage. Experimental results demonstrate that FactSim achieves significantly higher correlation with human judgments than current state-of-the-art automatic metrics, offering a more reliable proxy for human evaluation in assessing factual faithfulness.
📝 Abstract
We explore the need for more comprehensive and precise evaluation techniques for generative artificial intelligence (GenAI) in text summarization tasks, specifically in the area of opinion summarization. Traditional methods, which leverage automated metrics to compare machine-generated summaries from a collection of opinion pieces, e.g. product reviews, have shown limitations due to the paradigm shift introduced by large language models (LLM). This paper addresses these shortcomings by proposing a novel, fully automated methodology for assessing the factual consistency of such summaries. The method is based on measuring the similarity between the claims in a given summary with those from the original reviews, measuring the coverage and consistency of the generated summary. To do so, we rely on a simple approach to extract factual assessment from texts that we then compare and summarize in a suitable score. We demonstrate that the proposed metric attributes higher scores to similar claims, regardless of whether the claim is negated, paraphrased, or expanded, and that the score has a high correlation to human judgment when compared to state-of-the-art metrics.