Speaking in Self-Assessing Tongues: On the Verbalized Confidence of LLMs in Machine Translation

2026-06-15

AI Summary

This work addresses the unreliability of confidence estimates produced by large language models (LLMs) in machine translation and the limitations of existing unsupervised methods, which rely on internal model probabilities and require access to model internals. The authors propose five verbalization-based approaches that operate without any internal signals, establishing the first systematic framework for token-level confidence estimation independent of a model’s internal probabilities. Experimental results demonstrate that these verbalized confidence scores are nearly uncorrelated with internal probability signals yet achieve comparable performance in fine-grained error detection and confidence calibration tasks. The effectiveness varies across models, confirming the viability of verbalization as a reliable alternative for LLM self-evaluation and opening a new avenue for model-agnostic confidence assessment.

Abstract

The rapid rise in popularity of large language models (LLMs) for translation calls for a thorough study of the reliability of their confidence in their own outputs. Unlike many generation tasks, translation errors and confidence levels can be useful at different levels of granularity (tokens, words, or spans). Unsupervised approaches based on internal signals like predicted probabilities can be misleading because they reflect certainty among alternatives rather than correctness. In addition, they require access to such internal signals. Here, we devise five verbalized methods of extracting an LLM's per-token confidence without those shortcomings and compare their reliability with that of the model's internal signals of certainty. We evaluate reliability using two forms of alignment: fine-grained error detection and calibration. For both, internal and verbalized methods perform similarly, although results vary by model. Interestingly, we find little to no correlation between internal and verbalized methods.

Problem

• large language models
• machine translation
• confidence estimation
• verbalized confidence
• error detection

Innovation

• verbalized confidence
• large language models
• machine translation
• confidence calibration
• error detection