To Predict or Not to Predict? Towards reliable uncertainty estimation in the presence of noise

2026-03-07

AI Summary

This study addresses the challenge of unreliable predictions in multilingual text classification caused by noise and off-topic content. Focusing on the complex–simple sentence classification task, it systematically evaluates various uncertainty estimation methods—including Monte Carlo Dropout and softmax confidence—under low-resource settings, domain shifts, and multilingual conditions. The evaluation employs multidimensional metrics such as calibration, discriminative capacity, and decision threshold stability. Results demonstrate that Monte Carlo Dropout significantly outperforms conventional softmax-based approaches in noisy environments. Building on this insight, the authors propose a novel strategy that proactively abstains from high-risk predictions based on uncertainty estimates. Empirical validation on the Readme task shows that discarding the 10% most uncertain samples improves the macro F1 score from 0.81 to 0.85, underscoring the practical utility of this approach in enhancing model robustness and reliability.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Safety and Robustness
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

This study examines the role of uncertainty estimation (UE) methods in multilingual text classification under noisy and non-topical conditions. Using a complex-vs-simple sentence classification task across several languages, we evaluate a range of UE techniques against a range of metrics to assess their contribution to making more robust predictions. Results indicate that while methods relying on softmax outputs remain competitive in high-resource in-domain settings, their reliability declines in low-resource or domain-shift scenarios. In contrast, Monte Carlo dropout approaches demonstrate consistently strong performance across all languages, offering more robust calibration, stable decision thresholds, and greater discriminative power even under adverse conditions. We further demonstrate the positive impact of UE on non-topical classification: abstaining from predicting the 10\% most uncertain instances increases the macro F1 score from 0.81 to 0.85 in the Readme task. By integrating UE with trustworthiness metrics, this study provides actionable insights for developing more reliable NLP systems in real-world multilingual environments. See https://github.com/Nouran-Khallaf/To-Predict-or-Not-to-Predict

Problem

• uncertainty estimation
• multilingual text classification
• noise robustness
• non-topical classification
• reliable NLP

Innovation

• uncertainty estimation
• Monte Carlo dropout
• multilingual text classification
• robust NLP
• predictive abstention