Second Guess: Detecting Uncertainty Through Abstention and Answer Stability in Small Language Models

2026-05-24

AI Summary

This work addresses the tendency of small language models to produce confidently incorrect answers under uncertainty, stemming from a lack of effective uncertainty calibration. The authors propose a lightweight, parameter-free prompting method that introduces an explicit “I don’t know” option in multiple-choice question answering and leverages both answer stability and abstention behavior to assess model uncertainty. By uniquely integrating stability analysis with an abstention mechanism, this approach overcomes the degradation commonly observed in entropy-based methods when applied to fine-tuned or lower-performing models. Evaluated across four open-source small language models and four standard benchmarks, the method achieves up to a 10.81% reduction in overall risk and maintains a significant improvement of approximately 8% even on fine-tuned models.

Abstract

Large language models often generate confident but incorrect answers rather than abstaining when uncertain. This problem is particularly acute for small language models (SLMs), where computational constraints and autonomous operation amplify the need for reliable uncertainty detection. We propose _Second Guess_, a lightweight, parameter-free prompting technique for abstention in multiple-choice question answering (MCQA) that is well-suited for SLMs. Our key empirical insight is that models which truly know an answer will select it consistently, while uncertain models exhibit unstable behavior when an ``I don't know'' option is added. Evaluated on four open models (2B-8B parameters) and four benchmarks, Second Guess achieves the highest composite risk improvement of 10.81\%. Notably, it maintains an 8\% composite risk improvement on fine-tuned models where entropy-based methods degrade, and improves most for lower-performing models. All code and results required to reproduce this work is available in https://github.com/Mystic-Slice/second-guess

Problem

• uncertainty detection
• small language models
• abstention
• answer stability
• multiple-choice question answering

Innovation

• abstention
• uncertainty detection
• small language models
• answer stability
• prompting technique