Extract-and-Abstract: Unifying Extractive and Abstractive Summarization within Single Encoder-Decoder Framework

2024-09-18

arXiv.org

AI Summary

To address the error accumulation and high training costs arising from the long-standing separation of extractive and abstractive summarization, this paper proposes the ExtAbs paradigm: an end-to-end joint modeling framework within a unified encoder-decoder architecture. Its core innovation is a parameter-free saliency masking mechanism that dynamically modulates cross-attention weights to explicitly guide the decoder toward salient input segments. By eliminating conventional multi-stage training and auxiliary parameterized modules, ExtAbs enables synergistic optimization of extraction and abstraction. Built upon BART and PEGASUS backbones, ExtAbs achieves state-of-the-art extractive performance across CNN/DailyMail, XSum, and Newsroom benchmarks, while generating summaries on par with—or even surpassing—those of the original large models. These results validate the effectiveness and generalizability of lightweight, unified modeling for dual-purpose summarization.

Technology Category

• Natural Language Processing: Summarization
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Extract-then-Abstract is a naturally coherent paradigm to conduct abstractive summarization with the help of salient information identified by the extractive model. Previous works that adopt this paradigm train the extractor and abstractor separately and introduce extra parameters to highlight the extracted salients to the abstractor, which results in error accumulation and additional training costs. In this paper, we first introduce a parameter-free highlight method into the encoder-decoder framework: replacing the encoder attention mask with a saliency mask in the cross-attention module to force the decoder to focus only on salient parts of the input. A preliminary analysis compares different highlight methods, demonstrating the effectiveness of our saliency mask. We further propose the novel extract-and-abstract paradigm, ExtAbs, which jointly and seamlessly performs Extractive and Abstractive summarization tasks within single encoder-decoder model to reduce error accumulation. In ExtAbs, the vanilla encoder is augmented to extract salients, and the vanilla decoder is modified with the proposed saliency mask to generate summaries. Built upon BART and PEGASUS, experiments on three datasets show that ExtAbs can achieve superior performance than baselines on the extractive task and performs comparable, or even better than the vanilla models on the abstractive task.

Problem

• Unifying extractive and abstractive summarization in single model
• Reducing error accumulation from separate extractor-abstractor training
• Eliminating extra parameters through saliency mask attention method

Innovation

• Unified encoder-decoder framework for summarization
• Parameter-free saliency mask replaces attention mechanism
• Jointly performs extractive and abstractive tasks simultaneously