ESPnet-SDS: Unified Toolkit and Demo for Spoken Dialogue Systems

2025-03-11

AI Summary

Heterogeneous interfaces across speech dialogue systems hinder fair, cross-model evaluation. To address this, we propose the first open-source, unified evaluation toolkit supporting both cascaded and end-to-end paradigms. Built upon ESPnet, it integrates ASR, TTS, and LM modules and provides an interactive Gradio-based web interface for comparative analysis and automated benchmarking—including latency, semantic understanding, response coherence/diversity/relevance, speech intelligibility, and voice quality. Applying it to standard human-human dialogue datasets, we conduct the first systematic bottleneck analysis of end-to-end systems, revealing a 17% lower response diversity and a 0.8-point MOS degradation in synthesized speech versus cascaded baselines. The toolkit enables horizontal evaluation across six major system categories and is publicly deployed on Hugging Face Spaces, with its demo validated through over one thousand real-world user interactions.

Technology Category

• Natural Language Processing: Conversational AI/Dialog Systems
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Intelligent User Interfaces

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Advancements in audio foundation models (FMs) have fueled interest in end-to-end (E2E) spoken dialogue systems, but different web interfaces for each system makes it challenging to compare and contrast them effectively. Motivated by this, we introduce an open-source, user-friendly toolkit designed to build unified web interfaces for various cascaded and E2E spoken dialogue systems. Our demo further provides users with the option to get on-the-fly automated evaluation metrics such as (1) latency, (2) ability to understand user input, (3) coherence, diversity, and relevance of system response, and (4) intelligibility and audio quality of system output. Using the evaluation metrics, we compare various cascaded and E2E spoken dialogue systems with a human-human conversation dataset as a proxy. Our analysis demonstrates that the toolkit allows researchers to effortlessly compare and contrast different technologies, providing valuable insights such as current E2E systems having poorer audio quality and less diverse responses. An example demo produced using our toolkit is publicly available here: https://huggingface.co/spaces/Siddhant/Voice_Assistant_Demo.

Problem

• Unified toolkit for spoken dialogue systems comparison
• Automated evaluation metrics for system performance
• Comparison of cascaded and end-to-end dialogue systems

Innovation

• Open-source toolkit for unified web interfaces
• On-the-fly automated evaluation metrics provided
• Comparison of cascaded and E2E dialogue systems