BanglaSummEval: Reference-Free Factual Consistency Evaluation for Bangla Summarization

2026-02-18

AI Summary

This work proposes the first reference-free, end-to-end framework for evaluating factual consistency in low-resource languages, specifically targeting Bengali. Built upon a unified multilingual instruction-tuned model, the approach automatically generates questions from source documents and extracts corresponding answers from summaries, then assesses factual accuracy and content coverage by combining question importance weighting with BERTScore-Recall for semantic matching. Evaluated on a dataset of 300 human-written Bengali summaries in education and healthcare domains, the method demonstrates strong correlation with expert judgments (Pearson’s r = 0.694, Spearman’s ρ = 0.763), offering both interpretability and reliability without reliance on reference summaries.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Machine Learning: Evaluation and Analysis
• Knowledge Representation and Reasoning: Knowledge Representation Languages

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Evaluating factual consistency is essential for reliable text summarization, particularly in high-stakes domains such as healthcare and news. However, most existing evaluation metrics overlook Bangla, a widely spoken yet under-resourced language, and often depend on reference summaries. We introduce BanglaSummEval, a reference-free, question-answering-based framework for evaluating factual consistency in Bangla summarization. The proposed method assesses both factual accuracy and content coverage through automatically generated questions and answers derived from the source document and the summary. A single multilingual instruction-tuned language model handles question generation, question answering, candidate answer extraction, and question importance weighting. This unified design reduces system complexity and computational cost. To capture semantic consistency beyond surface-level overlap, we use BERTScore-Recall for answer comparison. We validate BanglaSummEval on 300 human-written summaries from educational and medical domains, demonstrating strong correlation with expert human judgments (Pearson's $r = 0.694$, Spearman's $ρ= 0.763$). By providing interpretable, step-wise diagnostics alongside reliable evaluation scores, BanglaSummEval offers a practical and transparent solution for factual consistency evaluation in low-resource language settings.

Problem

• factual consistency
• Bangla summarization
• reference-free evaluation
• low-resource language
• text summarization

Innovation

• reference-free evaluation
• factual consistency
• question-answering framework
• low-resource language
• multilingual instruction-tuned model