ERPBench: A State-Grounded Evaluation Paradigm for Computer-Use Agents in Enterprise Software

2026-09-15

AI Summary

研究解决了ERP系统中计算机使用代理的评估问题，通过引入ERPBench基准，基于真实ERP系统评估代理，并分析了代理在企业环境中的表现和失败模式。

Abstract

Computer-use agents that operate through screenshots and simulated actions are advancing rapidly, yet their evaluation remains anchored to general desktop and web tasks. Enterprise Resource Planning (ERP) systems run the finance, procurement, inventory, and customer operations of organizations worldwide, and pose distinct challenges for computer-use agents: dense interfaces, coordinated multi-step interactions, and errors that alter persistent business records rather than surfacing on screen. Existing enterprise benchmarks rely on proprietary platforms or on simulated approximations of such software. We introduce ERPBench, a benchmark that evaluates screenshot-only agents on a live and reproducible ERP system and scores each task against ground-truth values in its database. Beyond the benchmark, we present a production-grade harness that gates agent actions behind human approval for safe deployment, which ERPBench runs autonomously. Evaluating six closed and open-source agents, we demonstrate that strong general GUI performance does not transfer to enterprise reliability. Even when an agent reaches the right form and saves it, the stored record is often wrong: some agents save in up to 85% of runs but write the correct value in as few as 3%. We further characterize failure modes specific to enterprise workflows.

Problem

• Computer-use agents
• Enterprise Resource Planning (ERP)
• Evaluation benchmark

Innovation

• ERPBench
• state-grounded evaluation
• enterprise software
• production-grade harness