Assessing Reliability of BERT-Based Models on Question Answering Tasks
This study addresses the lack of systematic evaluation of reliability in question-answering models by proposing a novel framework that integrates Monte Carlo Dropout (MCD) with input semantic paraphrasing perturbations, requiring no modification to the standard inference pipeline. The authors conduct a comprehensive analysis of BERT, RoBERTa, ALBERT, and DistilBERT on the SQuAD and QuAC datasets. Experimental results demonstrate that MCD effectively captures prediction stability, with RoBERTa exhibiting the highest reliability, while ALBERT and DistilBERT show notably lower stability. This work not only validates MCD as a robust metric for model reliability but also establishes a new paradigm for evaluating the robustness of question-answering systems.