Learning to Summarize by Learning to Quiz: Adversarial Agentic Collaboration for Long Document Summarization

2025-09-25

AI Summary

Long-document summarization remains challenged by information loss, factual inconsistency, and poor coherence. To address these issues, we propose SummQ—a novel adversarial multi-agent collaborative framework centered on a question-answering–driven quiz agent that forms a closed-loop feedback mechanism with the summarization agent for iterative refinement. SummQ integrates five specialized roles: summarizer, reviewer, quiz generator, quiz reviewer, and quiz responder. It employs a hybrid evaluation strategy combining ROUGE, BERTScore, and LLM-as-a-Judge metrics. Extensive experiments across multiple benchmarks demonstrate that SummQ significantly outperforms state-of-the-art methods. Both automated and human evaluations confirm substantial improvements in information completeness, factual consistency, and textual coherence.

Technology Category

• Natural Language Processing: Summarization
• Multiagent Systems: Adversarial Agents
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Long document summarization remains a significant challenge for current large language models (LLMs), as existing approaches commonly struggle with information loss, factual inconsistencies, and coherence issues when processing excessively long documents. We propose SummQ, a novel adversarial multi-agent framework that addresses these limitations through collaborative intelligence between specialized agents operating in two complementary domains: summarization and quizzing. Our approach employs summary generators and reviewers that work collaboratively to create and evaluate comprehensive summaries, while quiz generators and reviewers create comprehension questions that serve as continuous quality checks for the summarization process. This adversarial dynamic, enhanced by an examinee agent that validates whether the generated summary contains the information needed to answer the quiz questions, enables iterative refinement through multifaceted feedback mechanisms. We evaluate SummQ on three widely used long document summarization benchmarks. Experimental results demonstrate that our framework significantly outperforms existing state-of-the-art methods across ROUGE and BERTScore metrics, as well as in LLM-as-a-Judge and human evaluations. Our comprehensive analyses reveal the effectiveness of the multi-agent collaboration dynamics, the influence of different agent configurations, and the impact of the quizzing mechanism. This work establishes a new approach for long document summarization that uses adversarial agentic collaboration to improve summarization quality.

Problem

• Addresses information loss and factual inconsistencies in long document summarization
• Improves summary coherence through adversarial multi-agent collaboration
• Enhances summarization quality using quiz-based quality checks

Innovation

• Adversarial multi-agent framework with summarization and quizzing agents
• Iterative refinement through quiz-based quality checks
• Collaborative intelligence between generators and reviewers