Confident in a Confidence Score: Investigating the Sensitivity of Confidence Scores to Supervised Fine-Tuning

2026-04-10

AI Summary

This study investigates how supervised fine-tuning degrades the correlation between language models’ confidence scores and output quality, thereby undermining uncertainty quantification. Through systematic analysis, we uncover the mechanisms by which fine-tuning—particularly via shifts in output distributions—decouples confidence from actual performance. Comparative experiments before and after fine-tuning, along with evaluations on downstream tasks, reveal that uncalibrated confidence metrics become significantly distorted post-fine-tuning, potentially leading to misleading reliability assessments in practical applications. Our findings underscore the necessity of task-specific validation for confidence measures and provide empirical grounding for the trustworthy deployment of fine-tuned language models.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Safety and Robustness
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Uncertainty quantification is a set of techniques that measure confidence in language models. They can be used, for example, to detect hallucinations or alert users to review uncertain predictions. To be useful, these confidence scores must be correlated with the quality of the output. However, recent work found that fine-tuning can affect the correlation between confidence scores and quality. Hence, we investigate the underlying behavior of confidence scores to understand its sensitivity to supervised fine-tuning (SFT). We find that post-SFT, the correlation of various confidence scores degrades, which can stem from changes in confidence scores due to factors other than the output quality, such as the output's similarity to the training distribution. We demonstrate via a case study how failing to address this miscorrelation reduces the usefulness of the confidence scores on a downstream task. Our findings show how confidence metrics cannot be used off-the-shelf without testing, and motivate the need for developing metrics which are more robust to fine-tuning.

Problem

• confidence score
• supervised fine-tuning
• uncertainty quantification
• hallucination detection
• output quality

Innovation

• confidence score
• supervised fine-tuning
• uncertainty quantification
• hallucination detection
• model calibration