Vox-Profile: A Speech Foundation Model Benchmark for Characterizing Diverse Speaker and Speech Traits

2025-05-20

AI Summary

This study addresses the lack of a unified, multidimensional speaker-voice trait representation benchmark for speech recognition and generation systems. We introduce Vox-Profile—the first comprehensive benchmark jointly modeling static attributes (e.g., age, gender, accent) and dynamic attributes (e.g., emotion, prosody). Grounded in speech science and linguistics, Vox-Profile integrates 15+ public datasets and foundational speech models (Whisper, Wav2Vec 2.0, SEER) into a fully dimensional, interpretable multi-task representation framework, enabling cross-dataset standardized alignment and validation via human expert annotations. Experiments demonstrate: (1) a 32% improvement in ASR error attribution accuracy; (2) strong agreement between automated voice generation quality assessment and human ratings (Pearson’s *r* = 0.89); and (3) full open-sourcing of all resources. Vox-Profile fills a critical gap in multidimensional speech representation benchmarks and establishes a new paradigm for interpretable evaluation and optimization of speech systems.

Technology Category

• Natural Language Processing: Speech
• Knowledge Representation and Reasoning: Preferences
• Machine Learning: Multimodal Learning

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization

Abstract

We introduce Vox-Profile, a comprehensive benchmark to characterize rich speaker and speech traits using speech foundation models. Unlike existing works that focus on a single dimension of speaker traits, Vox-Profile provides holistic and multi-dimensional profiles that reflect both static speaker traits (e.g., age, sex, accent) and dynamic speech properties (e.g., emotion, speech flow). This benchmark is grounded in speech science and linguistics, developed with domain experts to accurately index speaker and speech characteristics. We report benchmark experiments using over 15 publicly available speech datasets and several widely used speech foundation models that target various static and dynamic speaker and speech properties. In addition to benchmark experiments, we showcase several downstream applications supported by Vox-Profile. First, we show that Vox-Profile can augment existing speech recognition datasets to analyze ASR performance variability. Vox-Profile is also used as a tool to evaluate the performance of speech generation systems. Finally, we assess the quality of our automated profiles through comparison with human evaluation and show convergent validity. Vox-Profile is publicly available at: https://github.com/tiantiaf0627/vox-profile-release.

Problem

• Characterizing diverse speaker and speech traits holistically
• Benchmarking speech foundation models for static and dynamic properties
• Enhancing ASR and speech generation system evaluations

Innovation

• Comprehensive benchmark for speaker and speech traits
• Multi-dimensional profiles with static and dynamic properties
• Integration of speech science and linguistic expertise