RW-Voice-EQ Bench: A Real World Benchmark for Evaluating Voice AI Systems

2026-07-16

AI Summary

Current evaluations of spoken AI systems often rely on isolated metrics—such as recognition accuracy or textual quality—overlooking the acoustic characteristics and multidimensional nature inherent to speech. This work proposes the first comprehensive evaluation framework that integrates acoustic fidelity, expressiveness, interactivity, and robustness, establishing a real-world benchmark spanning text-to-speech (TTS), speech-to-speech translation (STS), spoken understanding (SU), and automatic speech recognition (ASR). The framework employs diverse data encompassing accents, emotions, background noise, and conversational contexts for fine-grained assessment. Findings reveal that system performance is highly dimension-dependent: TTS exhibits relative independence across dimensions, STS frequently neglects acoustic expressivity, SU shows inconsistent performance on paralinguistic tasks, and ASR exposes critical weaknesses under realistic conditions that conventional benchmarks fail to capture—thereby underscoring the inadequacy of single-metric evaluations.

Abstract

Current voice AI benchmarks typically evaluate isolated capabilities such as speech intelligibility, word error rate, or text-based dialogue quality, but they rarely test whether systems harness the acoustic information that distinguishes spoken language from its textual representation. To this end, we introduce the Real World Voice EQ Bench, a multidimensional benchmark for evaluating voice AI across text-to-speech (TTS), speech-to-speech (STS), speech understanding (SU), and automatic speech recognition (ASR). Our evaluations indicate that performance is highly dimension-specific. For TTS, naturalness, expressiveness, identity stability, and reliability are largely independent evaluation dimensions. For STS, access to audio does not guarantee use of vocal affect, and some agents remain largely transcript-driven. For SU, models perform unevenly across paralinguistic tasks. For ASR, real world accent, emotion, noise, and conversational conditions expose failures that are not captured by established clean-speech benchmarks. Together, these results show that voice AI should be evaluated as a profile of acoustic, expressive, interactional, and robustness capabilities rather than by a single aggregate score.

Problem

• voice AI
• benchmark
• acoustic information
• spoken language
• evaluation

Innovation

• voice AI benchmark
• acoustic information
• multidimensional evaluation
• paralinguistic understanding
• real-world robustness