Critic Experience Bank: Self-Evolving Step-Level Confidence Estimation for LLM Agents

2026-07-14

AI Summary

This work addresses the critical need for reliable step-level confidence estimation in large language model (LLM) agents, where single-step errors can lead to severe task failures. The authors propose a self-evolving critic framework that, for the first time, incorporates feedback on the consequences of executed actions into confidence assessment—without requiring ground-truth labels or additional training. By retrospectively evaluating outcomes to generate pseudo-labels, the method constructs and retrieves an experience memory bank, dynamically calibrating confidence through the integration of historical success and failure evidence when similar steps recur. Evaluated across three agent benchmarks and three backbone models, the approach significantly outperforms existing training-free baselines, achieving up to a 54% reduction in Expected Calibration Error (ECE) and setting new state-of-the-art results in both Brier score and AUC-based ranking performance.

Abstract

LLM agents act in external environments where each action changes the state that later decisions condition on, and where a single wrong step can waste interaction budget or trigger irreversible side effects long before the final failure is observed. Reliable deployment therefore requires \emph{step-level confidence estimation}: a calibrated probability that each proposed action is productive, available \emph{before} the action is executed. Existing LLM confidence estimators are designed to score a response from the given prompt, but agent confidence also depends on execution consequences: whether similar actions in similar situations actually advanced the task after the environment responded. We introduce the \method (\methodshort), a self-evolving critic framework in which an LLM critic accumulates evidence from its own past judgments and their observed consequences. After each trajectory, a hindsight LLM that sees the full execution feedback votes on whether each step was productive. The resulting pseudo-labels populate a memory bank from which related productive and unproductive experiences are retrieved into the critic's prompt whenever a similar step recurs. \methodshort requires no training and uses no ground truth step labels. Across three agent benchmarks and three critic backbones, \methodshort attains the best calibration (ECE and Brier) and ranking (AUC) in every dataset--critic combination, reducing ECE by up to $54\%$ relative to the strongest training-free baseline.

Problem

• step-level confidence estimation
• LLM agents
• execution consequences
• calibrated probability
• action productivity

Innovation

• step-level confidence estimation
• self-evolving critic
• experience bank
• LLM agents
• calibration