Reasoning with Confidence: Efficient Verification of LLM Reasoning Steps via Uncertainty Heads

2025-11-09

AI Summary

Existing reasoning verification methods—such as Preference Reward Models (PRMs)—suffer from high computational overhead, poor generalization, and reliance on costly human annotations. To address these limitations, we propose UHeads: a lightweight uncertainty prediction head that freezes the parameters of a large language model (LLM) and leverages its internal hidden states as confidence signals. UHeads trains a compact Transformer-based head (<10M parameters) via self-supervision or LLM-generated labels, enabling step-level reasoning verification without human annotation. Crucially, our work provides the first empirical evidence that LLM hidden states intrinsically encode reasoning reliability. Across diverse domains—including mathematical reasoning, planning, and commonsense question answering—UHeads matches or surpasses the performance of PRMs with 810× more parameters, while significantly improving verification efficiency, cross-domain generalization, and interpretability.

Technology Category

• Reasoning under Uncertainty: Uncertainty Representations
• Machine Learning: Calibration & Uncertainty Quantification
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Solving complex tasks usually requires LLMs to generate long multi-step reasoning chains. Previous work has shown that verifying the correctness of individual reasoning steps can further improve the performance and efficiency of LLMs on such tasks and enhance solution interpretability. However, existing verification approaches, such as Process Reward Models (PRMs), are either computationally expensive, limited to specific domains, or require large-scale human or model-generated annotations. Thus, we propose a lightweight alternative for step-level reasoning verification based on data-driven uncertainty scores. We train transformer-based uncertainty quantification heads (UHeads) that use the internal states of a frozen LLM to estimate the uncertainty of its reasoning steps during generation. The approach is fully automatic: target labels are generated either by another larger LLM (e.g., DeepSeek R1) or in a self-supervised manner by the original model itself. UHeads are both effective and lightweight, containing less than 10M parameters. Across multiple domains, including mathematics, planning, and general knowledge question answering, they match or even surpass the performance of PRMs that are up to 810x larger. Our findings suggest that the internal states of LLMs encode their uncertainty and can serve as reliable signals for reasoning verification, offering a promising direction toward scalable and generalizable introspective LLMs.

Problem

• Existing reasoning verification methods are computationally expensive or domain-specific
• Current approaches require large-scale human or model-generated annotations
• There is a need for lightweight automatic verification of LLM reasoning steps

Innovation

• UHeads estimate reasoning uncertainty from LLM states
• Automatically generates labels using larger or same LLM
• Lightweight UHeads match larger models with fewer parameters