Calibration Is Not Control: Why LLM-Agent Oversight Needs Intervention

2026-06-19

AI Summary

This work addresses a critical flaw in existing runtime supervision mechanisms for large language model agents, which erroneously rely on scalar risk scores—such as risk predictions—as proxies for intervention decisions without verifying whether interventions actually improve outcomes. To rectify this, the paper proposes using “intervention advantage,” defined as the expected utility gain of intervening over continuing execution, as the proper supervision objective, and formally introduces the notion of “target error” for the first time. The authors develop a prefix-branching protocol that enables counterfactual evaluation of alternative actions under identical trajectory prefixes, integrating action-conditioned control with scalar calibration decomposition. Experiments in the challenging ALFWorld interactive environment demonstrate that this approach reduces control regret from 0.506 to 0.110, whereas merely calibrating scalar scores improves prediction metrics but fails to lower control regret.

Abstract

Runtime oversight for LLM agents is commonly framed as scalar risk prediction: estimate failure likelihood, confidence, or uncertainty, then intervene once the score crosses a threshold. We argue that this framing targets the wrong object for control. The relevant question is not how likely the agent is to fail if it continues, but whether an available intervention would improve the outcome. Two trajectory prefixes can have the same risk estimate while requiring different actions, because one remains recoverable and the other does not. We formalize this mismatch as target error and identify intervention advantage, the expected utility gain from intervening rather than continuing, as the decision object for oversight. To measure this mismatch, we introduce prefix branching, a same-prefix counterfactual protocol that executes candidate actions from identical trajectory states. Across four benchmarks, action-conditioned control yields regime-dependent gains over scalar routing. In a calibration decomposition, recalibrating the same scalar score improves prediction metrics but leaves control regret unchanged, showing that calibration alone does not repair target error. A simple prefix-only action-conditioned controller substantially reduces regret in the strongest interactive regime, from 0.506 to 0.110 on ALFWorld. Gains shrink when interventions are weak or when scalar routing already preserves intervention-relevant information. These results suggest that LLM-agent oversight should move from calibrated risk scoring toward action-conditioned value estimation.

Problem

• LLM-agent oversight
• intervention advantage
• target error
• action-conditioned control
• calibration

Innovation

• intervention advantage
• target error
• prefix branching
• action-conditioned control
• calibration