DUAL: Diversity and Uncertainty Active Learning for Text Summarization

2025-03-02

AI Summary

Text summarization heavily relies on high-quality human annotations, yet existing active learning methods struggle to jointly optimize uncertainty and diversity—often being outperformed by random sampling. To address this, we propose DUAL, the first framework for summarization that jointly models both criteria: uncertainty is quantified via model prediction entropy, while diversity is captured through sentence embedding clustering and core-set selection. A dynamic weighting strategy balances these objectives during sample selection, mitigating the pitfalls of pure uncertainty-based sampling (which tends to select noisy instances) and pure diversity-driven sampling (which suffers from insufficient exploration). Evaluated on CNN/DailyMail and XSum with BART and PEGASUS, DUAL consistently outperforms diverse baselines, achieving an average ROUGE-L improvement of 1.8 points. This translates to substantially enhanced annotation efficiency and stronger model generalization.

Technology Category

• Natural Language Processing: Summarization
• Machine Learning: Active Learning
• Reasoning under Uncertainty: Uncertainty Representations

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Normalization, clustering, classification, and summarization of Web text
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

With the rise of large language models, neural text summarization has advanced significantly in recent years. However, even state-of-the-art models continue to rely heavily on high-quality human-annotated data for training and evaluation. Active learning is frequently used as an effective way to collect such datasets, especially when annotation resources are scarce. Active learning methods typically prioritize either uncertainty or diversity but have shown limited effectiveness in summarization, often being outperformed by random sampling. We present Diversity and Uncertainty Active Learning (DUAL), a novel algorithm that combines uncertainty and diversity to iteratively select and annotate samples that are both representative of the data distribution and challenging for the current model. DUAL addresses the selection of noisy samples in uncertainty-based methods and the limited exploration scope of diversity-based methods. Through extensive experiments with different summarization models and benchmark datasets, we demonstrate that DUAL consistently matches or outperforms the best performing strategies. Using visualizations and quantitative metrics, we provide valuable insights into the effectiveness and robustness of different active learning strategies, in an attempt to understand why these strategies haven't performed consistently in text summarization. Finally, we show that DUAL strikes a good balance between diversity and robustness.

Problem

• Improves active learning for text summarization by combining diversity and uncertainty.
• Addresses noisy sample selection in uncertainty-based methods and limited exploration in diversity-based methods.
• Demonstrates DUAL's consistent performance and robustness across various summarization models and datasets.

Innovation

• Combines uncertainty and diversity in active learning
• Iteratively selects representative and challenging samples
• Balances diversity and robustness in summarization