From Benchmarks to Business Impact: Deploying IBM Generalist Agent in Enterprise Production

2025-10-27

AI Summary

Enterprise intelligent agents face deployment bottlenecks including framework fragmentation, inefficient development, and the absence of production-grade evaluation standards. To address these, we propose CUGA—a general-purpose agent with a hierarchical planning-execution architecture and a strong analytical reasoning–driven decision mechanism, enabling flexible multimodal and cross-task adaptation. We introduce BPO-TA, the first 26-task benchmark targeting business process outsourcing (BPO) recruitment, enabling the first enterprise-scale validation of general agents’ feasibility and cost efficiency. CUGA achieves state-of-the-art performance on AppWorld and WebArena; in a recruitment pilot, its accuracy approaches that of task-specific agents while reducing development time and cost significantly. We open-source the CUGA system and distill key enterprise deployment pathways—including scalability, auditability, and security governance—alongside organizational implementation patterns.

Technology Category

• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Multiagent Systems: Multiagent Planning

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Agents are rapidly advancing in automating digital work, but enterprises face a harder challenge: moving beyond prototypes to deployed systems that deliver measurable business value. This path is complicated by fragmented frameworks, slow development, and the absence of standardized evaluation practices. Generalist agents have emerged as a promising direction, excelling on academic benchmarks and offering flexibility across task types, applications, and modalities. Yet, evidence of their use in production enterprise settings remains limited. This paper reports IBM's experience developing and piloting the Computer Using Generalist Agent (CUGA), which has been open-sourced for the community (https://github.com/cuga-project/cuga-agent). CUGA adopts a hierarchical planner--executor architecture with strong analytical foundations, achieving state-of-the-art performance on AppWorld and WebArena. Beyond benchmarks, it was evaluated in a pilot within the Business-Process-Outsourcing talent acquisition domain, addressing enterprise requirements for scalability, auditability, safety, and governance. To support assessment, we introduce BPO-TA, a 26-task benchmark spanning 13 analytics endpoints. In preliminary evaluations, CUGA approached the accuracy of specialized agents while indicating potential for reducing development time and cost. Our contribution is twofold: presenting early evidence of generalist agents operating at enterprise scale, and distilling technical and organizational lessons from this initial pilot. We outline requirements and next steps for advancing research-grade architectures like CUGA into robust, enterprise-ready systems.

Problem

• Transitioning agent prototypes to deployed systems delivering business value
• Addressing fragmented frameworks and lack of standardized evaluation practices
• Demonstrating generalist agent viability in enterprise production settings

Innovation

• Hierarchical planner-executor architecture for enterprise agents
• Open-sourced CUGA agent achieving state-of-the-art benchmark performance
• BPO-TA benchmark enabling enterprise-scale agent evaluation