Selected work

Representative Papers

RECOVER: Robust Entity Correction via agentic Orchestration of hypothesis Variants for Evidence-based Recovery

Mar 17, 2026

This work addresses the challenge of automatic speech recognition (ASR) errors caused by rare or missing domain-specific entities in specialized fields such as finance and healthcare, where post-processing correction is often difficult. The authors propose an agent-based entity correction framework that uniquely integrates ASR n-best hypotheses with tool-augmented large language models (LLMs), leveraging external entity retrieval and constraint-guided decoding for precise error correction. Several hypothesis fusion strategies—including 1-Best, Entity-Aware Select, ROVER Ensemble, and a novel LLM-Select—are introduced to substantially improve entity recall and accuracy. Evaluated across five datasets, the approach achieves relative reductions of 8%–46% in entity word error rate (E-WER) and up to a 22-percentage-point gain in entity recall, while maintaining overall word error rate (WER) stability.

0 citationsRead paper

Tool-Aware Planning in Contact Center AI: Evaluating LLMs through Lineage-Guided Query Decomposition

Feb 16, 2026

This work addresses the challenge that large language models struggle to effectively decompose complex user queries in contact center scenarios into executable, multi-step plans with clearly assigned tools—particularly when coordinating structured (e.g., Text2SQL) and unstructured (e.g., RAG) tools. To tackle this, the authors propose a tool-aware planning framework featuring a lineage-guided query decomposition method that generates dependency-aware steps amenable to parallel execution. They also introduce the first dual-mode evaluation benchmark for this task, comprising seven dimensions including tool-prompt alignment and query adherence. A systematic evaluation across 14 prominent models reveals significant performance degradation beyond four-step or composite queries. Claude-3-7-Sonnet achieves the highest overall score of 84.8%, while o3-mini attains a 49.75% single-match accuracy on “A+”-grade queries. Optimizing plan lineage substantially enhances step executability.

0 citationsRead paper

Counterfactual Fairness Evaluation of LLM-Based Contact Center Agent Quality Assurance System

Feb 16, 2026

This study addresses fairness concerns in large language models (LLMs) applied to customer service quality assurance, where biases related to agent identity, contextual cues, and behavioral styles—introduced during training—may lead to unfair evaluations. The authors construct the first counterfactual evaluation benchmark spanning 13 bias dimensions and systematically assess 18 LLMs on 3,000 real-world customer service dialogues. They propose a quantitative fairness metric combining Counterfactual Flip Rate (CFR) and Mean Absolute Score Difference (MASD), and conduct intervention analyses using fairness-aware prompts. Results reveal CFRs ranging from 5.4% to 13.0%, with contextual history inducing the most severe bias (CFR up to 16.4%). Explicit fairness prompting yields only marginal improvements, highlighting the critical influence of model scale and alignment on fairness outcomes.

0 citationsRead paper

Beyond IVR: Benchmarking Customer Support LLM Agents for Business-Adherence

Jan 02, 2026arXiv.org

Existing customer support systems struggle to flexibly handle complex, policy-driven tasks, and current evaluations of large language model (LLM) agents often overlook their ability to adhere to multi-step business rules. To address this gap, this work proposes JourneyBench—the first evaluation benchmark specifically designed to assess compliance with business policies in customer support scenarios. Leveraging a graph-based approach, JourneyBench generates diverse, realistic user journeys spanning three domains, comprising 703 distinct trajectories, and introduces a user journey coverage metric. By integrating static and dynamic prompting agent architectures (SPA/DPA) that explicitly model policy control flows, experiments demonstrate that the dynamic variant (DPA) significantly enhances policy adherence, enabling smaller models like GPT-4o-mini to outperform GPT-4o. These results underscore the critical role of structured policy orchestration in managing complex business processes.

0 citationsRead paper

Why Synthetic Isn't Real Yet: A Diagnostic Framework for Contact Center Dialogue Generation

Aug 25, 2025

Customer service dialogues are hindered by privacy constraints and scarcity of authentic corpora, impeding high-quality synthetic data generation and evaluation. To address their goal-oriented nature, role asymmetry, ASR-induced noise, and regulatory compliance requirements, we propose a multi-stage, feature-aware synthesis method grounded in intent summarization, topic flow modeling, and QA-formatted structuring. Furthermore, we introduce the first multilingual diagnostic framework tailored to customer service scenarios, comprising 18 linguistically and behaviorally grounded metrics. This framework enables fine-grained, reference-free assessment of disfluency, emotional consistency, and agent behavioral realism—marking the first such capability. Empirical analysis reveals systematic biases across existing generation methods: none fully approximate real-world dialogues, with pronounced deficiencies in emotional and behavioral modeling. Our diagnostic suite provides an interpretable, reproducible evaluation pathway for advancing synthetic dialogue systems.

0 citationsRead paper

Recent publications

Latest Papers

RECOVER: Robust Entity Correction via agentic Orchestration of hypothesis Variants for Evidence-based Recovery

Mar 17, 2026

This work addresses the challenge of automatic speech recognition (ASR) errors caused by rare or missing domain-specific entities in specialized fields such as finance and healthcare, where post-processing correction is often difficult. The authors propose an agent-based entity correction framework that uniquely integrates ASR n-best hypotheses with tool-augmented large language models (LLMs), leveraging external entity retrieval and constraint-guided decoding for precise error correction. Several hypothesis fusion strategies—including 1-Best, Entity-Aware Select, ROVER Ensemble, and a novel LLM-Select—are introduced to substantially improve entity recall and accuracy. Evaluated across five datasets, the approach achieves relative reductions of 8%–46% in entity word error rate (E-WER) and up to a 22-percentage-point gain in entity recall, while maintaining overall word error rate (WER) stability.

0 citationsRead paper

Tool-Aware Planning in Contact Center AI: Evaluating LLMs through Lineage-Guided Query Decomposition

Feb 16, 2026

This work addresses the challenge that large language models struggle to effectively decompose complex user queries in contact center scenarios into executable, multi-step plans with clearly assigned tools—particularly when coordinating structured (e.g., Text2SQL) and unstructured (e.g., RAG) tools. To tackle this, the authors propose a tool-aware planning framework featuring a lineage-guided query decomposition method that generates dependency-aware steps amenable to parallel execution. They also introduce the first dual-mode evaluation benchmark for this task, comprising seven dimensions including tool-prompt alignment and query adherence. A systematic evaluation across 14 prominent models reveals significant performance degradation beyond four-step or composite queries. Claude-3-7-Sonnet achieves the highest overall score of 84.8%, while o3-mini attains a 49.75% single-match accuracy on “A+”-grade queries. Optimizing plan lineage substantially enhances step executability.

0 citationsRead paper

Counterfactual Fairness Evaluation of LLM-Based Contact Center Agent Quality Assurance System

Feb 16, 2026

This study addresses fairness concerns in large language models (LLMs) applied to customer service quality assurance, where biases related to agent identity, contextual cues, and behavioral styles—introduced during training—may lead to unfair evaluations. The authors construct the first counterfactual evaluation benchmark spanning 13 bias dimensions and systematically assess 18 LLMs on 3,000 real-world customer service dialogues. They propose a quantitative fairness metric combining Counterfactual Flip Rate (CFR) and Mean Absolute Score Difference (MASD), and conduct intervention analyses using fairness-aware prompts. Results reveal CFRs ranging from 5.4% to 13.0%, with contextual history inducing the most severe bias (CFR up to 16.4%). Explicit fairness prompting yields only marginal improvements, highlighting the critical influence of model scale and alignment on fairness outcomes.

0 citationsRead paper

Beyond IVR: Benchmarking Customer Support LLM Agents for Business-Adherence

Jan 02, 2026arXiv.org

Existing customer support systems struggle to flexibly handle complex, policy-driven tasks, and current evaluations of large language model (LLM) agents often overlook their ability to adhere to multi-step business rules. To address this gap, this work proposes JourneyBench—the first evaluation benchmark specifically designed to assess compliance with business policies in customer support scenarios. Leveraging a graph-based approach, JourneyBench generates diverse, realistic user journeys spanning three domains, comprising 703 distinct trajectories, and introduces a user journey coverage metric. By integrating static and dynamic prompting agent architectures (SPA/DPA) that explicitly model policy control flows, experiments demonstrate that the dynamic variant (DPA) significantly enhances policy adherence, enabling smaller models like GPT-4o-mini to outperform GPT-4o. These results underscore the critical role of structured policy orchestration in managing complex business processes.

0 citationsRead paper

Why Synthetic Isn't Real Yet: A Diagnostic Framework for Contact Center Dialogue Generation

Aug 25, 2025

Customer service dialogues are hindered by privacy constraints and scarcity of authentic corpora, impeding high-quality synthetic data generation and evaluation. To address their goal-oriented nature, role asymmetry, ASR-induced noise, and regulatory compliance requirements, we propose a multi-stage, feature-aware synthesis method grounded in intent summarization, topic flow modeling, and QA-formatted structuring. Furthermore, we introduce the first multilingual diagnostic framework tailored to customer service scenarios, comprising 18 linguistically and behaviorally grounded metrics. This framework enables fine-grained, reference-free assessment of disfluency, emotional consistency, and agent behavioral realism—marking the first such capability. Empirical analysis reveals systematic biases across existing generation methods: none fully approximate real-world dialogues, with pronounced deficiencies in emotional and behavioral modeling. Our diagnostic suite provides an interpretable, reproducible evaluation pathway for advancing synthetic dialogue systems.

0 citationsRead paper