PrefSQA: Pairwise Preference Prediction for Speech Quality Assessment and the Critical Role of High Quality Datasets

2026-06-17

AI Summary

This study addresses the limitations of traditional mean opinion score (MOS)-based speech quality assessment, which suffers from high label noise and low reliability due to rater variability and test condition inconsistencies. To overcome these issues, the work proposes a novel pairwise preference prediction framework that operates without MOS labels, introducing an architecture integrating uncertainty-aware logistic output, a distortion-attention mechanism, and non-matching reference comparisons. A high-quality, multi-category preference dataset is also constructed, encompassing both simulated low-noise conditions and real human preferences. Experimental results demonstrate that the proposed method significantly outperforms existing baselines, thereby validating the efficacy of the preference prediction paradigm and highlighting the critical role of high-quality preference data in enhancing model performance.

Abstract

Mean opinion scores (MOS) are widely used for speech quality assessment, yet scalar labels are sensitive to rater variability and listening test differences. This introduces labeling noise, which limits the reliability of MOS prediction. Preference prediction reduces this variability as listeners compare signals directly, producing cleaner labels. We study MOS-free preference prediction and propose PrefSQA, which incorporates uncertainty-aware logits, an impairment attention head, and a module based on non-matching-reference comparisons. We use and refine five datasets, including MOS-derived and low-noise simulated sets with matching and non-matching content, experiment with human preference sets, and test on unseen data. Experiments show small improvements on MOS-derived data, while other sets reveal clear improvement over the baselines, highlighting the value of high-quality preference data and demonstrating the effectiveness of the proposed method.

Problem

• speech quality assessment
• preference prediction
• label noise
• mean opinion score
• high-quality datasets

Innovation

• preference prediction
• uncertainty-aware logits
• impairment attention
• non-matching-reference comparison
• speech quality assessment