🤖 AI Summary
Current evaluations of AI agents lack comprehensive assessment of cross-application coordination, autonomous API discovery, and policy adherence, failing to reflect the demands of real-world business automation. This work proposes the first unified benchmark that integrates cross-application workflow orchestration, autonomous API exploration, and regulatory compliance. Built upon the Zapier platform, the benchmark encompasses multi-scenario tasks spanning CRM, email, calendar, and other systems, requiring agents to autonomously discover APIs, follow multi-layered business rules, and execute cross-system data writes in realistic sales and marketing workflows. Evaluation relies solely on end-state correctness through an automated scoring mechanism. Experimental results reveal that even state-of-the-art models achieve success rates below 10%, highlighting a significant capability gap in deploying current AI systems for practical business automation.
📝 Abstract
Existing AI benchmarks for software automation rarely combine cross-application coordination, autonomous API discovery, and policy adherence. Real business workflows demand all three: a single task may span a CRM, inbox, calendar, and messaging platform - requiring the agent to find the right endpoints, follow a policy document, and write correct data to each system. To address this gap, we introduce AutomationBench, a benchmark for evaluating AI agents on cross-application workflow orchestration via REST APIs. Drawing on real workflow patterns from Zapier's platform, tasks span Sales, Marketing, Operations, Support, Finance, and HR domains. Agents must discover relevant endpoints themselves, follow layered business rules, and navigate environments with irrelevant and sometimes misleading records. Grading is programmatic and end-state only: whether the correct data ended up in the right systems. Even the best frontier models currently score below 10%. AutomationBench provides a challenging, realistic measure of where current models stand relative to the agentic capabilities businesses actually need.