Voice Quality Dimensions as Interpretable Primitives for Speaking Style for Atypical Speech and Affect

2025-05-27

AI Summary

This study addresses the challenge of modeling interpretable speech quality dimensions to characterize speaker styles in atypical and affective speech. We propose the first systematically defined set of seven phonatory quality primitives—e.g., clarity, glottalization, and breathiness—and train linear probes on frozen Wav2Vec 2.0 embeddings using the Speech Accessibility Project (SAP) dataset. Our method achieves strong interpretability and cross-domain generalizability: it attains state-of-the-art performance on SAP and, for the first time, enables zero-shot transfer across languages (English/Italian) and tasks (atypical vs. emotional speech), with average accuracy exceeding 82%. These results validate the framework’s efficacy as a universal, style-sensitive speech representation. The core contribution is the first interpretable, transferable, and multilingual-compatible speech quality dimension modeling framework.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Transparent, Interpretable, Explainable ML
• Cognitive Modeling & Cognitive Systems: Affective Computing

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Perceptual voice quality dimensions describe key characteristics of atypical speech and other speech modulations. Here we develop and evaluate voice quality models for seven voice and speech dimensions (intelligibility, imprecise consonants, harsh voice, naturalness, monoloudness, monopitch, and breathiness). Probes were trained on the public Speech Accessibility (SAP) project dataset with 11,184 samples from 434 speakers, using embeddings from frozen pre-trained models as features. We found that our probes had both strong performance and strong generalization across speech elicitation categories in the SAP dataset. We further validated zero-shot performance on additional datasets, encompassing unseen languages and tasks: Italian atypical speech, English atypical speech, and affective speech. The strong zero-shot performance and the interpretability of results across an array of evaluations suggests the utility of using voice quality dimensions in speaking style-related tasks.

Problem

• Modeling voice quality dimensions for atypical speech analysis
• Evaluating probes on intelligibility and speech characteristics
• Validating zero-shot performance across languages and tasks

Innovation

• Voice quality models for seven dimensions
• Probes trained on SAP dataset with embeddings
• Strong zero-shot performance across languages