Beyond IVR: Benchmarking Customer Support LLM Agents for Business-Adherence

2026-01-02

arXiv.org

AI Summary

Existing customer support systems struggle to flexibly handle complex, policy-driven tasks, and current evaluations of large language model (LLM) agents often overlook their ability to adhere to multi-step business rules. To address this gap, this work proposes JourneyBench—the first evaluation benchmark specifically designed to assess compliance with business policies in customer support scenarios. Leveraging a graph-based approach, JourneyBench generates diverse, realistic user journeys spanning three domains, comprising 703 distinct trajectories, and introduces a user journey coverage metric. By integrating static and dynamic prompting agent architectures (SPA/DPA) that explicitly model policy control flows, experiments demonstrate that the dynamic variant (DPA) significantly enhances policy adherence, enabling smaller models like GPT-4o-mini to outperform GPT-4o. These results underscore the critical role of structured policy orchestration in managing complex business processes.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Agentic search

Abstract

Traditional customer support systems, such as Interactive Voice Response (IVR), rely on rigid scripts and lack the flexibility required for handling complex, policy-driven tasks. While large language model (LLM) agents offer a promising alternative, evaluating their ability to act in accordance with business rules and real-world support workflows remains an open challenge. Existing benchmarks primarily focus on tool usage or task completion, overlooking an agent's capacity to adhere to multi-step policies, navigate task dependencies, and remain robust to unpredictable user or environment behavior. In this work, we introduce JourneyBench, a benchmark designed to assess policy-aware agents in customer support. JourneyBench leverages graph representations to generate diverse, realistic support scenarios and proposes the User Journey Coverage Score, a novel metric to measure policy adherence. We evaluate multiple state-of-the-art LLMs using two agent designs: a Static-Prompt Agent (SPA) and a Dynamic-Prompt Agent (DPA) that explicitly models policy control. Across 703 conversations in three domains, we show that DPA significantly boosts policy adherence, even allowing smaller models like GPT-4o-mini to outperform more capable ones like GPT-4o. Our findings demonstrate the importance of structured orchestration and establish JourneyBench as a critical resource to advance AI-driven customer support beyond IVR-era limitations.

Problem

• customer support
• business policy adherence
• LLM agents
• benchmarking
• user journey

Innovation

• JourneyBench
• policy adherence
• LLM agents
• User Journey Coverage Score
• Dynamic-Prompt Agent