DuplexWorld: Can voice agents help you get through the day?

2026-08-11

AI Summary

This work addresses the limitations of existing benchmarks in comprehensively evaluating voice agents’ performance in authentic everyday conversations, particularly their insufficient coverage of diverse dialogues and complex tasks. To bridge this gap, the authors propose the first multidimensional evaluation framework grounded in real-life scenarios, spanning six domains—such as banking, insurance, and travel—with 156 distinct situations encompassing over 350 hours of dialogue. The framework employs an end-to-end assessment methodology, integrating task success rate (Pass@1), turn-level quality scores, and DNSMOS to enable fine-grained analysis across conversational ability, task execution, and speech naturalness. Experimental results reveal that even state-of-the-art systems exhibit significant shortcomings across these core dimensions (Pass@1: 0.490, turn score: 0.653, DNSMOS: 3.378), underscoring substantial room for improvement in both reliability and naturalness of spoken interaction.

Abstract

Speech-to-speech (S2S) voice agents are increasingly being incorporated into enterprise for customer care and as daily companions for consumers owing to the ease of the conversational modality over text. However, existing benchmarks fail to holistically evaluate voice agents along axes that really matter and are shaped as tests of agentic tool calling against a database. We believe they fail to adequately account for the diversity of conversational dialogue that mundane activities introduce and further, never test how faithfully an agent can assist on tasks that move beyond database manipulation. To tackle this DuplexWorld introduces six worlds where voice agents are especially useful: banking, insurance, travel, healthcare and logistics, and Pathfinding. Agents are evaluated on eleven different types of conversations across 156 scenarios (350+ hours of conversation), each testing conversational and analytical capability to varying degrees. Through extensive evaluation comprising agentic, conversational and speech-naturalness metrics, we show that even the best voice agents leave substantial room for improvement on all 3 axes (Pass@1: 0.490, turn-taking: 0.653, DNSMOS: 3.378). We perform extensive analysis on agentic v conversational performance, world- and conversation type-wise performance, failure modes exploring the explore v exploit lens for Pathfinding conversations and voice agent reliability over all six worlds.

Problem

• voice agents
• speech-to-speech
• conversational dialogue
• benchmarking
• task assistance

Innovation

• voice agents
• speech-to-speech
• conversational AI
• benchmarking
• agentic evaluation