When are likely answers right? On Sequence Probability and Correctness in LLMs

2026-06-25

AI Summary

This study investigates the relationship between sequence probability and answer correctness in large language models, aiming to clarify when high-probability outputs are more likely to be correct. Through systematic evaluation across four granularities—spanning diverse decoding strategies, multiple models, and benchmark datasets—and complemented by conditional probability analysis and empirical validation, the work provides the first quantitative assessment of the alignment between sequence probability and correctness. The findings reveal that while sequence probability exhibits moderate predictive power within a fixed dataset, it becomes unreliable under decoding optimization or when generating multiple responses from the same prompt. These results delineate the practical boundaries of sequence probability as a correctness proxy, offering theoretical insights and actionable guidance for designing decoding strategies and enabling model self-improvement.

Abstract

Many decoding methods for large language models can be understood as shifting probability mass toward outputs that are more likely under the model, either locally at the token level or globally at the sequence level. Therefore, their success depends on a fundamental question: when does sequence probability, that is, the conditional probability of a continuation given a prompt, actually align with correctness? In this paper, we set out to quantify this relationship across decoding methods, models, and benchmarks at four levels: across decoding methods, across hyperparameters within a method, across prompt-answer pairs within a dataset, and across repeated responses to the same prompt. We find that higher sequence probability is often predictive of correctness across prompt-answer pairs within a fixed dataset. However, this relationship does not generally transfer to decoding decisions: increasing sequence probability by changing hyperparameters or methods does not reliably improve accuracy. Further, sequence probability is not a good indicator of correctness for responses to the same prompt. These findings clarify when decoding can and cannot be expected to improve correctness, and provide practical guidance for decoding, self-consistency, and verifier-free self-improvement.

Problem

• sequence probability
• correctness
• large language models
• decoding methods
• answer reliability

Innovation

• sequence probability
• correctness
• decoding methods
• large language models
• self-consistency