NVMOS: Non-Verbal Vocalization Quality Assessment in Speech

2026-06-14

AI Summary

This study addresses the critical gap in existing speech quality assessment methods, which commonly overlook the perceptual quality of non-linguistic vocalizations—such as laughter and sighs—and lack dedicated evaluation frameworks. To bridge this gap, the authors construct NV-MOS, the first subjective rating dataset encompassing both synthesized and naturally recorded non-linguistic vocalizations across multiple systems. They further propose NVMOS, a specialized evaluation model featuring a local event attention module that effectively captures salient acoustic characteristics. Experimental results demonstrate that NVMOS achieves or even surpasses inter-expert agreement levels in quality prediction and significantly outperforms general-purpose multimodal large language models. Moreover, the study reveals substantial discrepancies between these large models and human judgments, underscoring their current inadequacy as substitutes for human assessment in this domain.

Abstract

Non-verbal vocalizations (NVs), such as laughter, sighs, and coughs, are important acoustic cues for emotion and intent. Existing speech quality assessment methods typically focus on overall naturalness, while non-verbal TTS evaluations mainly examine whether a target NV appears with the correct type and position. However, the perceptual quality of NV events themselves remains underexplored. To address this gap, we construct an NV-MOS dataset containing outputs from multiple NV-TTS systems and naturally occurring NV samples, with ratings collected from three acoustic experts on a perceptual quality scale. We further analyze audio-capable multimodal large language models such as Gemini and find clear inconsistencies between their scores and expert ratings. These results suggest that general-purpose multimodal models cannot reliably replace human judgments for NV quality assessment. We then propose NVMOS, to our knowledge the first model that can reliably predict the perceptual quality of NV events in speech. Experimental results show that, with a local NV-event focusing module, NVMOS reaches expert-level or stronger agreement with human MOS.

Problem

• Non-verbal vocalizations
• Speech quality assessment
• Perceptual quality
• NV-TTS
• MOS

Innovation

• Non-verbal vocalization
• Speech quality assessment
• Perceptual quality
• Multimodal LLM evaluation
• NVMOS