A Self Consistency Based Reranking for Narrative Question Answering

2026-06-14

AI Summary

This work addresses the susceptibility of single-decoding approaches in narrative question answering to generation stochasticity, which often yields incomplete or inconsistent answers. To mitigate this issue, the authors propose a self-consistency reranking framework that requires no modification to the underlying model architecture. The method generates multiple candidate answers and reranks them based on semantic similarity, selecting the optimal response through consensus among the candidates. Evaluated on the NarrativeQA dataset, the approach demonstrates substantial performance gains when applied to both pretrained and fine-tuned language models: accuracy for Pegasus-Large improves from 72.50% to 87.07%, while FLAN-T5-Base achieves 86.66%. These results highlight the framework’s effectiveness in enhancing both the robustness and accuracy of narrative QA systems.

Abstract

Narrative question answering (NQA) is a challenging task in natural language processing that requires models to understand long textual contexts, capture relationships across events, and generate coherent responses. Despite recent advances in pretrained language models, most existing approaches rely on a single decoding output during inference, making them sensitive to generation variability and often resulting in incomplete or inconsistent answers .To address this limitation, we propose a self-ensemble Self-Consistency-Based reranking framework for narrative question answering. The proposed method generates multiple candidate answers for each story-question pair and selects the final answer based on semantic agreement among the generated responses. This allows the model to explore diverse answer formulations while improving robustness through consensus-based selection without requiring modifications to the underlying architecture .The framework combines pretrained and fine-tuned language generation with multi-answer inference and similarity-based reranking. We evaluate the proposed approach on the NarrativeQA dataset using multiple models, including FLAN-T5 (Base and Small) and Pegasus-Large, under both baseline and fine-tuned settings .Experimental results demonstrate that the proposed method consistently improves performance across all models. In particular, FLAN-T5-Base achieves the best overall performance, improving from 82.32% to 86.66% (+4.34%) when combined with self-ensemble inference. Additionally, the largest improvement is observed with Pegasus-Large, which increases from 72.50% to 87.07% (+14.57%), highlighting the effectiveness of the proposed strategy.

Problem

• Narrative Question Answering
• Answer Consistency
• Generation Variability
• Inference Robustness
• Long-form QA

Innovation

• Self-Consistency
• Reranking
• Narrative Question Answering
• Multi-answer Inference
• Semantic Agreement