Scaling Laws for Agent Harnesses via Effective Feedback Compute

2026-05-28

AI Summary

This work addresses the limitation of existing test-time scaling analyses that rely on raw computational costs—such as token counts or tool invocations—which fail to distinguish informative, effective feedback from redundant or unstable interactions, leading to inaccurate performance predictions. To overcome this, we propose Effective Feedback Compute (EFC), a novel scaling coordinate that accounts for feedback only when it is informative, non-redundant, and actually influences subsequent decisions, normalized by task-specific requirements to enable cross-task comparison. Through trajectory-level analysis, feedback validity assessment, and controlled experiments, we develop Oracle-EFC and Estimated-EFC estimators. In both synthetic and real-world tasks, EFC achieves correlation coefficients (R²) up to 0.99—substantially outperforming raw metrics (R² ≈ 0.3–0.4). Moreover, under fixed compute budgets, improving feedback quality alone elevates task success rates from 0.27 to 0.90, demonstrating that scaling efficacy hinges more on feedback quality than sheer computational volume.

Abstract

Agent harnesses increasingly determine the performance of language-model systems by deciding how models call tools, receive feedback, verify intermediate states, store memory, and revise solutions. Yet current test-time scaling analyses often parameterize this process by raw expenditure -- tokens, tool calls, operations, wall time, or cost -- which does not distinguish useful feedback from redundant or unstable interaction. We introduce \emph{Effective Feedback Compute} (EFC), a trace-level scaling coordinate that credits feedback only when it is informative, valid, non-redundant, and retained for subsequent decisions, and we normalize it by task demand when comparing tasks with different feedback requirements. Across synthetic controllable tasks, executable code tasks, real benchmark traces, held-out splits, and a prospective validation batch, EFC-based coordinates consistently predict failure rates better than raw-compute baselines and a strong multivariate SAS baseline. In controlled scaling, raw tokens and tool calls explain limited variation ($R^2=0.33$ and $0.42$), SAS reaches $0.88$, while Oracle-EFC and Estimated-EFC reach $0.94$ and Oracle-EFC/$D_{\mathrm{task}}$ reaches $0.99$. Matched-budget interventions show that improving feedback quality raises success from $0.27$ to $0.90$ while raw cost and tool calls are fixed. On mixed real traces, NRS-EFC/$D_{\mathrm{task}}$ reaches $R^2=0.92$ while raw compute has near-zero or negative fit, and it remains the best predictor in a prospective holdout ($R^2=0.85$). These results suggest that harness scaling is governed less by how much computation is spent than by how efficiently raw budget is converted into durable, task-sufficient feedback.

Problem

• Scaling Laws
• Effective Feedback Compute
• Agent Harnesses
• Test-time Scaling
• Feedback Quality

Innovation

• Effective Feedback Compute
• Agent Harness
• Scaling Laws
• Feedback Efficiency
• Test-Time Scaling