Selecting N-lowest scores for training MOS prediction models

2025-06-23

AI Summary

In speech quality assessment, human subjective ratings are often dominated by low-quality segments, rendering conventional Mean Opinion Scores (MOS) inadequate for capturing true perceptual quality. To address this, we propose N_low-MOS—a metric that replaces global MOS with the mean of the N lowest human ratings—aligning more closely with humans’ “weakest-link” sensitivity in quality evaluation. We integrate N_low-MOS into the training and optimization of MOSNet on the VCC2018 and BVCC datasets. Experiments demonstrate substantial improvements: linear correlation coefficient (LCC) and Spearman rank correlation coefficient (SRCC) increase by 5.2% and 6.8%, respectively, and discriminative capability for degraded regions in voice conversion systems is notably enhanced. This work provides the first systematic empirical validation of the dominant role of low-scoring segments in subjective assessment and establishes a psychologically grounded evaluation paradigm for speech quality modeling.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Evaluation and Analysis
• Search and Optimization: Evaluation and Analysis

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization

Abstract

The automatic speech quality assessment (SQA) has been extensively studied to predict the speech quality without time-consuming questionnaires. Recently, neural-based SQA models have been actively developed for speech samples produced by text-to-speech or voice conversion, with a primary focus on training mean opinion score (MOS) prediction models. The quality of each speech sample may not be consistent across the entire duration, and it remains unclear which segments of the speech receive the primary focus from humans when assigning subjective evaluation for MOS calculation. We hypothesize that when humans rate speech, they tend to assign more weight to low-quality speech segments, and the variance in ratings for each sample is mainly due to accidental assignment of higher scores when overlooking the poor quality speech segments. Motivated by the hypothesis, we analyze the VCC2018 and BVCC datasets. Based on the hypothesis, we propose the more reliable representative value N_low-MOS, the mean of the $N$-lowest opinion scores. Our experiments show that LCC and SRCC improve compared to regular MOS when employing N_low-MOS to MOSNet training. This result suggests that N_low-MOS is a more intrinsic representative value of subjective speech quality and makes MOSNet a better comparator of VC models.

Problem

• Identify key speech segments humans focus on for MOS ratings
• Propose N_low-MOS as a more reliable speech quality metric
• Improve MOS prediction model accuracy using N_low-MOS training

Innovation

• Uses N-lowest scores for MOS prediction
• Focuses on low-quality speech segments
• Improves LCC and SRCC metrics