Doomed from the Start: Early Abort of LLM Agent Episodes via a Recall-Controlled Probe Cascade

2026-07-07

AI Summary

This work addresses the inefficiency of large language model (LLM) agents that often persist along inevitably failing trajectories in multi-step tasks, wasting substantial computational resources. The authors propose the first multi-stage early-exit framework that provides a user-specified global recall guarantee. By deploying lightweight probes to monitor hidden states—shown to signal failure earlier than observable actions—the framework implements a cascaded abort mechanism to terminate unproductive executions proactively. Key innovations include a distribution-agnostic calibration gating strategy, joint optimization of per-round recall budgets, and a theoretical characterization of sample complexity under high-recall regimes. Experiments on TextCraft demonstrate that the method reduces inference compute by 47.1% for Qwen-2.5-7B and 37.2% for Llama-3.2-3B at 90% recall, achieving 1.6–1.7× the performance of the best single-threshold baseline.

Abstract

Large language model (LLM) agents solving multi-step tasks frequently commit to trajectories that are doomed to fail, yet continue to consume substantial inference compute before the failure becomes observable. We show that failure is predictable early from the agent's internal representations: lightweight per-round probes on hidden activations anticipate eventual episode failure as early as the first interaction round, where scorers reading only the agent's observable behavior are barely better than chance. We turn this signal into a practical abort cascade: one distribution-free calibrated gate per round, with per-round recall budgets jointly searched so that eventually-successful episodes survive all gates at a user-specified global rate; this episode-level guarantee is the one that matters in deployment, since false-abort risk accumulates across gates. Across two agent models on TextCraft, the cascade meets every recall target from 90% to 97% and, at the 90% target, saves 47.1% +/- 10.3% (Qwen-2.5-7B) and 37.2% +/- 8.8% (Llama-3.2-3B) of inference compute, 1.6--1.7x the best single-gate policy. An otherwise-identical cascade reading only behavior saves roughly half as much, and adding behavioral features to the probe yields no further gain: the hidden states capture what behavior reveals. Finally, we characterize the sample complexity of certifying high recall targets, telling practitioners which recall promises their data can, and provably cannot, back. The code will be released soon.

Problem

• LLM agents
• early failure prediction
• inference compute waste
• multi-step tasks
• episode abort

Innovation

• early failure prediction
• recall-controlled cascade
• hidden activation probing
• inference compute savings
• sample complexity certification