EnterpriseClawBench: Benchmarking Agents from Real Workplace Sessions

2026-06-22

AI Summary

This study addresses the lack of realistic benchmarks for evaluating AI agents in authentic enterprise work environments, which hinders comprehensive assessment of their capabilities in complex office settings. The authors introduce the first structured evaluation suite derived from real-world enterprise agent conversations, comprising 852 tasks—each specifying a prompt, role category, skill subcategory, hard constraints, and semantic scoring criteria. They further propose a multidimensional evaluation protocol encompassing model-framework combinations, artifact delivery, visual quality, cost, runtime efficiency, and skill transferability. Experimental results reveal that even the best-performing configuration (Codex + GPT-5.5) achieves only a score of 0.663, underscoring the significant limitations of current agents on enterprise-grade tasks and affirming the necessity and value of developing multidimensional, real-scenario evaluation benchmarks.

Abstract

Enterprise agents increasingly operate inside workspaces: they read heterogeneous files, invoke tools, and deliver business artifacts. We introduce EnterpriseClawBench, an enterprise agent benchmark constructed from proprietary, real-world agent sessions. Starting from a large archive of workplace sessions, the EnterpriseClawBench produces 852 reproducible tasks, each paired with recovered fixtures, rewritten prompts, role classes, skill subclasses, hard rules, and semantic rubrics. Because the sessions contain internal enterprise content, we do not release the benchmark data; instead, our reusable contribution is the construction and evaluation protocol. On EnterpriseClawBench, the best configuration reaches only 0.663 (Codex with GPT-5.5). These results show that enterprise agent evaluation must report harness--model combinations, artifact delivery, visual quality, cost, runtime, and skill-transfer behavior, rather than collapsing performance into a single score. Code: https://github.com/FrontisAI/EnterpriseClawBench

Problem

• enterprise agents
• benchmarking
• real-world sessions
• artifact delivery
• evaluation protocol

Innovation

• Enterprise Agents
• Real-world Benchmarking
• Task Reproducibility
• Multi-dimensional Evaluation
• Evaluation Protocol