LongSumEval: Question-Answering Based Evaluation and Feedback-Driven Refinement for Long Document Summarization

2026-04-27

AI Summary

Existing evaluation methods for long-document summarization exhibit weak correlation with human judgments and lack interpretability and actionable guidance for improvement. This work proposes a unified question-answering–based framework that, for the first time, aligns evaluation with generation: by constructing structured question-answer pairs, it analyzes factual consistency and coverage gaps to produce interpretable scores and actionable feedback, which in turn drive a training-free self-refinement mechanism—effectively realizing “evaluation as instruction.” The approach significantly improves correlation with human judgments across seven benchmark datasets and demonstrably enhances summary quality.

Abstract

Evaluating long document summaries remains the primary bottleneck in summarization research. Existing metrics correlate weakly with human judgments and produce aggregate scores without explaining deficiencies or guiding improvement, preventing effective refinement in applications requiring verifiable accuracy. We introduce LongSumEval, a unified framework bridging evaluation and generation through structured question-answering feedback. The framework operationalizes summary quality as answerability and factual alignment of question-answer pairs, generating interpretable scores and actionable feedback that identifies coverage gaps and factual inconsistencies. This resolves the misalignment where evaluation operates independently of generation objectives. Meta-evaluation of our QA-based evaluation module across seven benchmarks demonstrates substantially stronger agreement with human judgments compared to established metrics. Structured feedback enables significant quality improvements through self-refinement without retraining. By demonstrating that evaluation feedback can serve as executable instructions for generation, this work establishes a generalizable paradigm for aligning assessment with improvement, with direct implications for controllable text generation requiring verifiable accuracy and transparent quality control. All code and datasets will be released in GitHub for reproducibility.

Problem

• long document summarization
• evaluation metrics
• human judgment alignment
• factual consistency
• summary refinement

Innovation

• question-answering based evaluation
• feedback-driven refinement
• long document summarization
• factual consistency
• interpretable feedback