VoxMind: An End-to-End Agentic Spoken Dialogue System

2026-04-17

AI Summary

This work addresses the limitations of existing end-to-end spoken dialogue systems, which struggle with complex real-world tasks due to insufficient agent capabilities. To overcome this, the authors propose a fully agent-capable end-to-end spoken dialogue system that incorporates a “think-before-speak” mechanism, internalizing structured reasoning as a prerequisite for response generation. An asynchronous multi-agent architecture is introduced to enable dynamic tool retrieval and collaborative reasoning, effectively decoupling tool scale from reasoning latency. Trained on the 470-hour AgentChat dataset, the system achieves a task completion rate of 74.57% on spoken-agent benchmarks—substantially outperforming Gemini-2.5-Pro’s 34.88%—while maintaining high-quality general conversational performance.

Technology Category

• Natural Language Processing: Conversational AI/Dialog Systems
• Multiagent Systems: Agent Communication
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Recent end-to-end spoken dialogue models enable natural interaction. However, as user demands become increasingly complex, models that rely solely on conversational abilities often struggle to cope. Incorporating agentic capabilities is therefore essential: by enabling tool use, these models can extend their knowledge boundaries and better solve real-world tasks. Yet, existing research has largely concentrated on core perception and generation, with comparatively limited exploration of such tool-augmented extensions. To bridge this gap, we present VoxMind, an integrated framework designed to equip end-to-end spoken dialogue models with comprehensive agentic abilities. Leveraging our curated 470-hour AgentChat dataset, we incorporate a "Think-before-Speak" mechanism, enabling the model to internalize structured reasoning as a critical prerequisite for planning and response generation. Furthermore, to mitigate latency bottlenecks caused by large-scale tool integration, we propose a Multi-Agent Dynamic Tool Management architecture. By asynchronously delegating retrieval tasks to an auxiliary agent aligned with the main model's reasoning trajectory, this system effectively decouples inference latency from toolset size. Experimental results confirm that VoxMind achieves significant improvements in agent performance: compared with strong baselines, the task completion rate increases from 34.88% to 74.57%, outperforming Gemini-2.5-Pro on spoken agent tasks while preserving general conversational quality. The source code and associated data are publicly available at https://github.com/MM-Speech/VoxMind.

Problem

• spoken dialogue system
• agentic capabilities
• tool use
• end-to-end modeling
• task completion

Innovation

• agentic spoken dialogue system
• Think-before-Speak mechanism
• Multi-Agent Dynamic Tool Management
• tool-augmented reasoning
• end-to-end speech interaction