EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents

2026-05-13

AI Summary

Existing benchmarks for evaluating spoken-language agents struggle to simultaneously assess realistic dialogue generation and capture agent-specific failure modes. This work proposes the first end-to-end evaluation framework that generates high-quality test samples through dynamic, multi-turn spoken conversations between simulated agents, augmented with an automatic validation mechanism and a test suite incorporating accent and noise perturbations. The framework introduces two composite metrics—EVA-A (task completion and speech fidelity) and EVA-X (dialogue experience)—combined with pass@1, pass@k, and pass^k evaluation protocols to enable architecture-agnostic assessment. Experiments across twelve state-of-the-art systems reveal that no system exceeds 0.5 on both EVA-A and EVA-X under pass@1; the gap between peak and median reliable performance reaches 0.44; and robustness degrades by up to 0.314 under perturbation testing.

Abstract

Voice agents, artificial intelligence systems that conduct spoken conversations to complete tasks, are increasingly deployed across enterprise applications. However, no existing benchmark jointly addresses two core evaluation challenges: generating realistic simulated conversations, and measuring quality across the full scope of voice-specific failure modes. We present EVA-Bench, an end-to-end evaluation framework that addresses both. On the simulation side, EVA-Bench orchestrates bot-to-bot audio conversations over dynamic multi-turn dialogues, with automatic simulation validation that detects user simulator error and appropriately regenerates conversations before scoring. On the measurement side, EVA-Bench introduces two composite metrics: EVA-A (Accuracy), capturing task completion, faithfulness, and audio-level speech fidelity; and EVA-X (Experience), capturing conversation progression, spoken conciseness, and turn-taking timing. Both metrics apply to different agent architectures, enabling direct cross-architecture comparison. EVA-Bench includes 213 scenarios across three enterprise domains, a controlled perturbation suite for accent and noise robustness, and pass@1, pass@k, pass^k measurements that distinguish peak from reliable capability. Across 12 systems spanning all three architectures, we find: (1) no system simultaneously exceeds 0.5 on both EVA-A pass@1 and EVA-X pass@1; (2) peak and reliable performance diverge substantially (median pass@k - pass^k gap of 0.44 on EVA-A); and (3) accent and noise perturbations expose substantial robustness gaps, with effects varying across architectures, systems, and metrics (mean up to 0.314). We release the full framework, evaluation suite, and benchmark data under an open-source license.

Problem

• voice agents
• evaluation benchmark
• simulated conversations
• voice-specific failure modes
• speech fidelity

Innovation

• voice agent evaluation
• end-to-end benchmark
• bot-to-bot simulation
• composite metrics
• robustness perturbation