ClawForge: Generating Executable Interactive Benchmarks for Command-Line Agents

2026-05-13

AI Summary

Existing benchmarks for interactive agents struggle to simultaneously ensure scalability and effectively evaluate performance under realistic workflow conditions involving state conflicts—such as partial, stale, or contradictory prior states. To address this gap, this work proposes ClawForge, the first executable command-line workflow framework that systematically supports evaluation under pre-existing state conflicts. ClawForge enables reproducible task construction through scenario templates, state initialization, reference trajectories, and validators, while abandoning strict trajectory matching in favor of stepwise assessment based on normalized final states and observable side effects. The accompanying ClawForge-Bench comprises 17 scenarios; evaluations reveal that even the best-performing model achieves only a 45.3% strict accuracy, with all models exhibiting errorful state replacement rates below 17%. Critically, the tendency to proactively inspect existing states emerges as a key determinant of performance disparities.

Abstract

Interactive agent benchmarks face a tension between scalable construction and realistic workflow evaluation. Hand-authored tasks are expensive to extend and revise, while static prompt evaluation misses failures that only appear when agents operate over persistent state. Existing interactive benchmarks have advanced agent evaluation significantly, but most initialize tasks from clean state and do not systematically test how agents handle pre-existing partial, stale, or conflicting artifacts. We present \textbf{ClawForge}, a generator-backed benchmark framework for executable command-line workflows under state conflict. The framework compiles scenario templates, grounded slots, initialized state, reference trajectories, and validators into reproducible task specifications, and evaluates agents step by step over persistent workflow surfaces using normalized end state and observable side effects rather than exact trajectory matching. We instantiate this framework as the ClawForge-Bench (17 scenarios, 6 ability categories). Results across seven frontier models show that the best model reaches only 45.3% strict accuracy, wrong-state replacement remains below 17\% for all models, and the widest model separation (17% to 90%) is driven by whether agents inspect existing state before acting. Partial-credit and step-efficiency analyses further reveal that many failures are near-miss closures rather than early breakdowns, and that models exhibit qualitatively different failure styles under state conflict.

Problem

• interactive agent benchmarks
• state conflict
• command-line workflows
• persistent state
• executable benchmarks

Innovation

• state conflict
• executable benchmark
• command-line agents
• persistent workflow
• trajectory-agnostic evaluation