Assessing LLMs for Zero-shot Abstractive Summarization Through the Lens of Relevance Paraphrasing

2024-06-06

arXiv.org

AI Summary

This work investigates the robustness of large language models (LLMs) in zero-shot abstractive summarization. We identify a critical vulnerability: minor input perturbations induce substantial output instability in generated summaries. To systematically probe this issue, we propose “relevance-based rewriting”—a novel paradigm that selects salient sentences via sentence-level relevance scoring and applies minimal, semantics-preserving perturbations to construct unlabeled, fine-tuning-free robustness probes. Our approach is the first to jointly integrate controllable input perturbation with quantitative consistency measurement of summaries. We conduct comprehensive evaluations across four representative LLMs—GPT-3.5-Turbo, Llama-2, Mistral, and Dolly—on four benchmark datasets (CNN/DailyMail, XSum, etc.). Results reveal a sharp degradation in summary consistency under minimal perturbations, with ROUGE-L scores fluctuating by 12–28%, exposing severe robustness deficiencies in current zero-shot summarization. This work establishes a reproducible evaluation benchmark and analytical framework for trustworthy abstractive summarization.

Technology Category

• Natural Language Processing: Summarization
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Large Language Models (LLMs) have achieved state-of-the-art performance at zero-shot generation of abstractive summaries for given articles. However, little is known about the robustness of such a process of zero-shot summarization. To bridge this gap, we propose relevance paraphrasing, a simple strategy that can be used to measure the robustness of LLMs as summarizers. The relevance paraphrasing approach identifies the most relevant sentences that contribute to generating an ideal summary, and then paraphrases these inputs to obtain a minimally perturbed dataset. Then, by evaluating model performance for summarization on both the original and perturbed datasets, we can assess the LLM's one aspect of robustness. We conduct extensive experiments with relevance paraphrasing on 4 diverse datasets, as well as 4 LLMs of different sizes (GPT-3.5-Turbo, Llama-2-13B, Mistral-7B, and Dolly-v2-7B). Our results indicate that LLMs are not consistent summarizers for the minimally perturbed articles, necessitating further improvements.

Problem

• Large Language Models
• Stability
• Accuracy in Summarization

Innovation

• Correlation Paraphrasing
• Stability Assessment
• Large Language Models