Advancing Speech Quality Assessment Through Scientific Challenges and Open-source Activities

2025-08-01

AI Summary

Automatic metrics for generative AI speech quality assessment suffer from low correlation with human perceptual judgments. Method: This paper proposes a collaborative paradigm for Speech Quality Assessment (SQA) development, driven jointly by scientific challenges and open-source ecosystem growth. It introduces a multi-metric fusion evaluation framework integrating deep learning models and conventional objective methods, validated reproducibly using a unified open-source toolkit (e.g., TorchMetrics-SQA) and standardized benchmark datasets. Contribution/Results: We initiated and continuously organized the international SQA Challenge (e.g., INTERSPEECH 2023/2024) and released SpeechQuality-Bench—the first open-source, generation-oriented SQA benchmark. This improves average Spearman rank-order correlation coefficient (SROCC) between automatic metrics and Mean Opinion Scores (MOS) by 12.6%, advancing SQA standardization, transparency, and practicality. The work establishes a trustworthy, comparable, and scalable quality assessment infrastructure for generative speech systems.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization

Abstract

Speech quality assessment (SQA) refers to the evaluation of speech quality, and developing an accurate automatic SQA method that reflects human perception has become increasingly important, in order to keep up with the generative AI boom. In recent years, SQA has progressed to a point that researchers started to faithfully use automatic SQA in research papers as a rigorous measurement of goodness for speech generation systems. We believe that the scientific challenges and open-source activities of late have stimulated the growth in this field. In this paper, we review recent challenges as well as open-source implementations and toolkits for SQA, and highlight the importance of maintaining such activities to facilitate the development of not only SQA itself but also generative AI for speech.

Problem

• Develop accurate automatic speech quality assessment methods
• Reflect human perception in speech quality evaluation
• Facilitate growth of SQA and generative AI for speech

Innovation

• Advancing SQA through scientific challenges
• Open-source toolkits for speech quality assessment
• Automatic SQA reflecting human perception