CRMArena: Understanding the Capacity of LLM Agents to Perform Professional CRM Tasks in Realistic Environments

2024-11-04

arXiv.org

AI Summary

Existing CRM evaluation benchmarks fail to capture real-world business complexity, hindering the integration and validation of AI agents in professional settings. Method: We introduce CRMArena—the first industrial-grade CRM workflow benchmark—featuring nine realistic tasks across three roles (service agent, analyst, manager), grounded in 16 highly interdependent object types and latent-variable modeling to encode intricate business logic and regulatory constraints. It uniquely integrates domain expert knowledge with latent-variable formalization and evaluates agents via dual paradigms: ReAct prompting and structured function calling, under high-fidelity object-relational modeling and dynamic data distribution simulation. Contribution/Results: CRMArena systematically exposes critical LLM agent limitations in rule adherence and structured function invocation. Experiments show state-of-the-art LLM agents achieve only 40% task completion under ReAct and 55% under function calling—highlighting the stringent demands of real-world CRM on robustness, compliance, and structured operational fidelity.

Technology Category

• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Modeling other Agents
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Customer Relationship Management (CRM) systems are vital for modern enterprises, providing a foundation for managing customer interactions and data. Integrating AI agents into CRM systems can automate routine processes and enhance personalized service. However, deploying and evaluating these agents is challenging due to the lack of realistic benchmarks that reflect the complexity of real-world CRM tasks. To address this issue, we introduce CRMArena, a novel benchmark designed to evaluate AI agents on realistic tasks grounded in professional work environments. Following guidance from CRM experts and industry best practices, we designed CRMArena with nine customer service tasks distributed across three personas: service agent, analyst, and manager. The benchmark includes 16 commonly used industrial objects (e.g., account, order, knowledge article, case) with high interconnectivity, along with latent variables (e.g., complaint habits, policy violations) to simulate realistic data distributions. Experimental results reveal that state-of-the-art LLM agents succeed in less than 40% of the tasks with ReAct prompting, and less than 55% even with function-calling abilities. Our findings highlight the need for enhanced agent capabilities in function-calling and rule-following to be deployed in real-world work environments. CRMArena is an open challenge to the community: systems that can reliably complete tasks showcase direct business value in a popular work environment.

Problem

• Evaluating AI agents in realistic CRM tasks
• Lack of benchmarks for CRM complexity
• Enhancing agent capabilities for real-world deployment

Innovation

• Realistic CRM task evaluation
• Multi-persona task design
• Latent variable simulation