Enhancing Factuality through Consensus and Consistency in Summarization Using Minimum Bayes Risk Decoding

2026-05-28

AI Summary

This work proposes a novel re-ranking approach to enhance the factuality of generated summaries—defined as their consistency with the source document—by integrating, for the first time, a consensus mechanism based on Minimum Bayes Risk (MBR) decoding with a factuality-oriented evaluation of source-document consistency. The method jointly re-ranks multiple candidate summaries, effectively balancing diversity preservation with improved factual accuracy. Experimental results demonstrate that the proposed approach achieves strong performance on automatic metrics and significantly outperforms existing systems in human evaluations, thereby validating its effectiveness and novelty in advancing summary factuality.

Abstract

Improving the quality of model-generated summaries, especially factuality, the accuracy of a summary with respect to its source content, remains a challenge. While reranking could select the optimal output from multiple generated candidates, it is limited to only using the source as guidance, resulting in unreliable summaries. To address this limitation, we propose ConSUM that reranks candidate summaries by considering two factors: consistency to the source document and consensus among the other candidates. Consensus is established using Minimum Bayes Risk (MBR) decoding over the set of generated summaries, while ensuring consistency by employing factuality-aware metrics that compare the summary against the source. Rigorous testing demonstrates that our system is competitive with existing methods, with human evaluations further confirming that its generated summaries are preferred over those from other systems. Our code is available at https://github.com/naist-nlp/ConSUM .

Problem

• factuality
• summarization
• reranking
• consistency
• consensus

Innovation

• Minimum Bayes Risk decoding
• factuality
• consensus
• consistency
• summary reranking