Can one size fit all?: Measuring Failure in Multi-Document Summarization Domain Transfer

2025-03-20

AI Summary

This work investigates the failure mechanisms of multi-document summarization (MDS) models under zero-shot cross-domain transfer (news → scientific → conversational domains). We systematically evaluate four dominant paradigms—end-to-end pretraining, chunked summarization, extractive-then-generative, and GPT-style reasoning—using both human judgment and automated metrics. For the first time, we unify and quantify domain-transfer failure across three dimensions: reference similarity, summary quality, and factual consistency. Our analysis reveals that failure stems primarily from semantic structural mismatch between training paradigms and target-domain discourse properties—not merely from distributional shift. Moreover, standard automatic metrics (e.g., ROUGE, BERTScore) exhibit significant miscalibration in cross-domain settings. Based on these findings, we propose principled metric calibration strategies and introduce the first cross-domain benchmark specifically designed for robustness evaluation of MDS models. This benchmark provides empirical grounding and methodological guidance for developing generalizable, domain-agnostic summarization systems.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Natural Language Processing: Summarization
• Application Domains: Humanities & Computational Social Science

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success

Abstract

Abstractive multi-document summarization (MDS) is the task of automatically summarizing information in multiple documents, from news articles to conversations with multiple speakers. The training approaches for current MDS models can be grouped into four approaches: end-to-end with special pre-training ("direct"), chunk-then-summarize, extract-then-summarize, and inference with GPT-style models. In this work, we evaluate MDS models across training approaches, domains, and dimensions (reference similarity, quality, and factuality), to analyze how and why models trained on one domain can fail to summarize documents from another (News, Science, and Conversation) in the zero-shot domain transfer setting. We define domain-transfer"failure"as a decrease in factuality, higher deviation from the target, and a general decrease in summary quality. In addition to exploring domain transfer for MDS models, we examine potential issues with applying popular summarization metrics out-of-the-box.

Problem

• Evaluates MDS models across domains and training approaches.
• Analyzes failure in zero-shot domain transfer for summarization.
• Examines issues with standard summarization metrics application.

Innovation

• Evaluates MDS models across diverse training approaches.
• Analyzes domain transfer failure in zero-shot settings.
• Examines issues with standard summarization metrics.