Are You Sure You're Sure? On the Impact of Instruction Tuning on Confidence and Lexical Diversity

2026-08-13

AI Summary

This study investigates how instruction tuning influences confidence expression and lexical diversity in the generated rationales of language models on question-answering tasks. By systematically comparing matched base and instruction-tuned models across multiple QA benchmarks—using confidence calibration metrics, lexical diversity measures, and controlled analyses—it reveals that instruction tuning consistently induces overconfidence and degrades likelihood calibration, even when accuracy gains are negligible. Furthermore, it significantly reduces semantic diversity in rationales across samples, while surface-level lexical diversity exhibits inconsistent changes. These findings highlight previously underappreciated, implicit effects of instruction tuning on model reasoning behavior, offering a new perspective for its refinement and evaluation.

Abstract

Instruction-tuned language models achieve strong performance across a range of generation tasks, but have also recently been shown to exhibit verbalized overconfidence. In question answering, verbalized model overconfidence may be associated with the consistency of the generated supporting rationales. In this paper, we study whether corresponding changes in the lexical diversity of generated answer rationales accompany changes in model confidence induced by instruction tuning. We evaluate three matched base and instruction-tuned models across question-answering benchmarks and find that instruction tuning consistently alters answer confidence, despite limited changes in predictive accuracy and decreases in likelihood-based calibration. Secondly, we observe a non-uniform effect of instruction tuning on rationale diversity: cross-rationale diversity consistently decreases, whereas surface-level lexical diversity varies in both direction and magnitude across models and benchmarks. Finally, we find that these differences persist after controlling for answer selection and rationale length, confirming that confidence and rationale diversity capture distinct effects of instruction tuning.

Problem

• instruction tuning
• model confidence
• lexical diversity
• question answering
• rationale generation

Innovation

• instruction tuning
• model confidence
• lexical diversity
• rationale generation
• language model calibration