🤖 AI Summary
This work addresses the unreliability of confidence estimation in current large language model (LLM) judge systems, which stems from the absence of token-level log probabilities and structured outputs. The authors propose a single-call confidence estimation method that requires no additional inference passes, leveraging decomposed chain-of-verification reasoning traces to extract three structured signals: Step-Verdict Alignment, Claim-Level Margin, and Evidence Grounding Score. The approach integrates decomposed reasoning, Platt-scaled logistic regression, and a lightweight natural language inference (NLI) model—used in place of regular expressions for evidence extraction. Experiments demonstrate substantial improvements over conventional logprob-based baselines, achieving AUROC scores of 0.66–0.91 on GPT-family models, elevating Qwen-family performance from poorly calibrated ranges of 0.32–0.49 to 0.56–0.70, and reaching 0.73–0.88 in production systems, with strong cross-model transferability.
📝 Abstract
LLM-as-Judge systems are widely deployed for automated evaluation, yet practitioners lack reliable methods to know when a judge's verdict should be trusted. Token log-probabilities, the standard post-hoc confidence signal, are unavailable for many commercial LLMs and, even when accessible, saturate above 0.999 with structured JSON output.
We introduce VERDI (VERification-Decomposed Inference), a method that extracts confidence from the reasoning trace a structured judge already produces, with no additional inference calls. VERDI decomposes each verification-style evaluation into sub-checks and derives three structural signals: Step-Verdict Alignment, Claim-Level Margin, and Evidence Grounding Score. We combine them with Platt-scaled logistic regression.
On three public benchmarks, VERDI achieves AUROC 0.72-0.91 on GPT-4.1-mini and 0.66-0.80 on GPT-5.4-mini. On Qwen3.5-4B/9B/27B, where answer-token logprobs are anti-calibrated (higher confidence on errors, AUROC 0.32-0.49), VERDI achieves 0.56-0.70. We additionally validate on a production system with eight rubrics (AUROC 0.73-0.88 on factual rubrics), demonstrate cross-model transfer (AUROC 0.66-0.69), and show that a 33M-parameter NLI (Natural Language Inference) model provides a scalable alternative to regex extraction.