Beyond Confidence: Rethinking Self-Assessments for Performance Prediction in LLMs

2026-05-08

AI Summary

Current large language models often rely on confidence scores that exhibit inconsistency and excessive optimism, limiting their reliability in predicting task performance. This work proposes the first systematic integration of psychological appraisal theory into self-evaluation for large language models, introducing a multidimensional framework encompassing six dimensions—including effort and ability. Through prompt engineering, the authors elicited multidimensional self-assessment signals from twelve models across thirty-eight tasks and quantitatively evaluated their predictive validity. Results demonstrate that the effort and ability dimensions consistently outperform conventional confidence metrics, with effort yielding more robust and less overconfident predictions. Moreover, the optimal predictive dimension varies significantly across task types, underscoring the effectiveness and adaptability of the proposed multidimensional self-evaluation mechanism.

Abstract

Large Language Models (LLMs) are increasingly used in settings where reliable self-assessment is critical. Assessing model reliability has evolved from using probabilistic correctness estimates to, more recently, eliciting verbalized confidence. Confidence, however, has been shown to be an inconsistent and overoptimistic predictor of model correctness. Drawing on cognitive appraisal theory, a framework from human psychology that decomposes self-evaluation into multiple components, we propose a multidimensional perspective on model self-assessment. We elicit six appraisal-based dimensions of self-assessment, alongside confidence, and evaluate their utility for predicting model failure across 12 LLMs and 38 tasks spanning eight domains. We find that competence-related appraisal dimensions, particularly effort and ability, consistently match or outperform confidence across most settings. Effort additionally yields less overoptimistic estimates that remain stable across model sizes. In contrast, affective dimensions provide marginally predictive signals. Furthermore, the most informative dimension varies systematically with task characteristics: effort is most predictive for reasoning-intensive tasks, while ability and confidence dominate on retrieval-oriented tasks. Broadly, our findings indicate that structured multidimensional self-assessment is a promising approach to improving the reliability and safety of language model deployment across diverse real-world settings.

Problem

• self-assessment
• confidence
• performance prediction
• large language models
• reliability

Innovation

• multidimensional self-assessment
• cognitive appraisal theory
• effort
• ability
• performance prediction