Document Summarization with Conformal Importance Guarantees

2025-09-24

AI Summary

To address the lack of theoretical guarantees for critical information retention in automatic summarization for high-stakes domains (e.g., healthcare, law), this paper proposes the first importance-preserving summarization framework integrated with conformal prediction. Methodologically, it pioneers the application of conformal prediction to summarization—requiring no distributional assumptions—and achieves adaptive threshold calibration via sentence-level importance scoring combined with a small calibration set. The framework is model-agnostic, supports black-box large language models, and enables tunable precision–recall trade-offs. Contributions include: (1) theoretical guarantees on coverage and recall for user-specified critical content; and (2) empirical validation across major summarization benchmarks, demonstrating consistent effectiveness and robustness in diverse scenarios, with significant improvements in reliability of critical information preservation.

Technology Category

• Natural Language Processing: Summarization
• Machine Learning: Calibration & Uncertainty Quantification
• Data Mining & Knowledge Management: Data Visualization & Summarization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Normalization, clustering, classification, and summarization of Web text

Abstract

Automatic summarization systems have advanced rapidly with large language models (LLMs), yet they still lack reliable guarantees on inclusion of critical content in high-stakes domains like healthcare, law, and finance. In this work, we introduce Conformal Importance Summarization, the first framework for importance-preserving summary generation which uses conformal prediction to provide rigorous, distribution-free coverage guarantees. By calibrating thresholds on sentence-level importance scores, we enable extractive document summarization with user-specified coverage and recall rates over critical content. Our method is model-agnostic, requires only a small calibration set, and seamlessly integrates with existing black-box LLMs. Experiments on established summarization benchmarks demonstrate that Conformal Importance Summarization achieves the theoretically assured information coverage rate. Our work suggests that Conformal Importance Summarization can be combined with existing techniques to achieve reliable, controllable automatic summarization, paving the way for safer deployment of AI summarization tools in critical applications. Code is available at https://github.com/layer6ai-labs/conformal-importance-summarization.

Problem

• Providing reliable guarantees for critical content inclusion in automatic summarization
• Developing importance-preserving summary generation with rigorous coverage guarantees
• Enabling extractive document summarization with user-specified critical content coverage

Innovation

• Conformal prediction for importance-preserving summary generation
• Calibrating thresholds on sentence-level importance scores
• Model-agnostic method integrating with existing black-box LLMs