Latent Confidence Alignment for LLM Self-Assessment

2026-06-20

AI Summary

This work addresses the limitation in current large language models (LLMs) where confidence calibration often disregards item difficulty, making it difficult to discern whether high confidence stems from genuine self-assessment or artifacts of the generation process. To tackle this, the study introduces a novel framework that incorporates item difficulty into self-evaluation by leveraging the Rasch model to construct a latent ability space. Adopting a metacognitive perspective, it proposes the Latent Confidence Alignment Error (LCAE) metric to quantify the consistency between a model’s self-reported confidence and the error probability implied by its latent ability and item difficulty. Evaluation across 20 LLMs on a medical dataset demonstrates that this approach significantly enhances self-evaluation reliability without compromising task performance, while also uncovering a link between model reliability and reasoning cost.

Abstract

Confidence calibration in large language models (LLMs) is commonly evaluated by comparing predicted confidence with observed accuracy. However, such approaches do not model item difficulty, making it difficult to interpret discrepancies and to determine whether model confidence reflects genuine self-assessment or is merely a byproduct of the response generation process. To address this, we adopt a Rasch model-based latent ability framework and a metacognitive perspective, and propose Latent Confidence Alignment Error (LCAE) to measure the consistency between model self-assessment and the latent error probability implied by model ability and item difficulty. We further incorporate item difficulty as an external signal with a reasoning mechanism. Experiments on a medical-domain dataset with 20 models show that the proposed approach improves self-assessment quality without affecting model ability, and reveals an association between reliability and inference cost.

Problem

• confidence calibration
• self-assessment
• item difficulty
• large language models
• latent ability

Innovation

• Latent Confidence Alignment
• Rasch model
• self-assessment
• item difficulty
• confidence calibration