Estimating Uncertainty from Reasoning: A Large-Scale Study of Multi- and Crosslingual MCQA Performance in LLMs

2026-07-07

AI Summary

This study addresses the limited understanding of uncertainty estimation (UE) for large language models (LLMs) in multilingual settings, particularly for low-resource languages, as existing research predominantly focuses on English. The authors conduct a large-scale evaluation of nine open-box and closed-box UE methods across 22 languages using human-annotated question-answering data and chain-of-thought reasoning prompts, deliberately avoiding noisy proxies such as LLM-as-a-judge or embedding-based scores. Their analysis reveals, for the first time, that generating reasoning chains in English significantly enhances UE performance for low-resource languages, suggesting that the language of generation outweighs the language of the input question in importance. Furthermore, smaller models are better suited for probabilistic open-box methods, while larger models excel in self-expressive closed-box approaches. The work also provides practical calibration guidelines that effectively narrow the performance gap between high- and low-resource languages.

Abstract

Uncertainty estimation (UE) enables LLM-powered systems to recognize when to abstain, yet existing research has predominantly focused on English. We present the first large-scale evaluation of UE methods across 22 languages, spanning high-, mid-, and low-resource settings. Using two human-curated Q\&A datasets, we compare open and closed box UE methods (nine in total) across different model sizes and architectures while eliciting long-form reasoning, avoiding LLM-as-a-judge and embedding-based scoring, which can introduce evaluation noise. We report three main actionable findings. First, we find that prompting models to reason in English while keeping questions in low-resource languages substantially improves UE performance, suggesting that comprehension of low-resource languages is largely intact, and that the reliability bottleneck lies in generation rather than understanding. Second, prompting models to reason in English closes the UE performance gap between low and high-resource languages, demonstrating that generation language matters more than the question language. Third, the choice of UE method should depend on model scale: at smaller scales, open-box probability-based methods outperform alternatives; at larger scales, closed-box self-verbalized uncertainty becomes superior. Finally, we provide an analysis of threshold selection for selective prediction, offering guidance on calibrating abstention in multilingual settings.

Problem

• Uncertainty Estimation
• Multilingual
• Low-resource Languages
• Large Language Models
• MCQA

Innovation

• Uncertainty Estimation
• Multilingual LLMs
• Crosslingual Reasoning
• Selective Prediction
• Large Language Models