Rethinking Mean Opinion Scores in Speech Quality Assessment: Aggregation through Quantized Distribution Fitting

2025-06-23

AI Summary

Traditional speech quality assessment (SQA) relies on discrete Mean Opinion Score (MOS) annotations (1–5), causing perceptual information loss and limiting model discriminability. To address this, we propose a distributional MOS modeling framework: we treat discrete MOS ratings as quantized observations of an underlying continuous latent variable reflecting true perceptual quality, and directly regress the full probability distribution—e.g., a Gaussian mixture—over this latent space using a neural network. Quality prediction is derived from the distribution’s mode rather than its integer-rounded mean. We introduce a quantization-aware likelihood-based loss for end-to-end optimization of distribution fitting. Integrating this approach into MOSNet by replacing its scalar output head yields consistent improvements across benchmarks—including DNSMOS and VoiceMOS—with gains of +0.08–0.12 in PLCC and SRCC, alongside enhanced robustness. This work establishes the first paradigm shift in SQA from discrete score regression to continuous perceptual distribution modeling.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Distributed Search

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Speech quality assessment (SQA) aims to evaluate the quality of speech samples without relying on time-consuming listener questionnaires. Recent efforts have focused on training neural-based SQA models to predict the mean opinion score (MOS) of speech samples produced by text-to-speech or voice conversion systems. This paper targets the enhancement of MOS prediction models' performance. We propose a novel score aggregation method to address the limitations of conventional annotations for MOS, which typically involve ratings on a scale from 1 to 5. Our method is based on the hypothesis that annotators internally consider continuous scores and then choose the nearest discrete rating. By modeling this process, we approximate the generative distribution of ratings by quantizing the latent continuous distribution. We then use the peak of this latent distribution, estimated through the loss between the quantized distribution and annotated ratings, as a new representative value instead of MOS. Experimental results demonstrate that substituting MOSNet's predicted target with this proposed value improves prediction performance.

Problem

• Enhance MOS prediction models in speech quality assessment
• Address limitations of conventional 1-5 scale MOS annotations
• Propose quantized distribution fitting for better score aggregation

Innovation

• Quantized distribution fitting for MOS aggregation
• Modeling annotators' latent continuous scores
• Peak estimation from latent distribution improves MOS prediction