ProPS: Prompted Profile Synthesis for Natural Language-Conditioned Speaker Embedding Distributions

2026-07-06

AI Summary

This work addresses the limitation of existing speaker embedding extractors, which can only describe observed speech and cannot generate controllable speaker characteristics from natural language prompts. To bridge this gap, the authors propose the ProPS framework, which encodes textual descriptions into sentence embeddings and employs a mixture density network to model a Gaussian mixture distribution in the x-vector space, enabling semantic-driven generation of speaker embeddings. This approach is the first to produce structurally coherent and attribute-controllable speaker representations conditioned on natural language, effectively aligning linguistic semantics with speaker attribute spaces. Experimental results demonstrate that the generated x-vectors exhibit high consistency across attributes such as age, gender, accent, and prosody, with both negative log-likelihood and attribute classification accuracy confirming the validity of the learned distributions.

Abstract

Speaker embeddings, or x-vectors, are widely used to represent speaker identity and speaker-related attributes, but existing embedding extractors are typically descriptive rather than generative: they map an observed speech segment to an x-vector, which is then used for downstream applications. We introduce ProPS, Prompted Profile Synthesis, a framework for generating distributions of speaker embeddings conditioned on natural language prompts such as "a thirties male speaker with an Indian accent". ProPS converts human-written profile descriptions into sentence embeddings and uses a mixture density network trained on a large-scale dataset to predict a Gaussian mixture model in the x-vector space. The model is trained by maximizing the likelihood that real speaker embeddings match the requested profile, and its generated distributions are evaluated by negative log-likelihood on held-out x-vectors and by attribute classification accuracies on sampled synthetic x-vectors. Experiments show that ProPS produces profile-conditioned distributions and generates x-vectors that preserve requested speaker attributes such as age, gender, accent, and prosodic characteristics. This design enables controllable speaker-profile synthesis for speech generation systems like Text-To-Speech (TTS) or Voice Conversion (VC) while anchoring generated distributions in observed speaker-embedding structure.

Problem

• speaker embedding
• natural language conditioning
• generative modeling
• profile synthesis
• x-vector

Innovation

• Prompted Profile Synthesis
• speaker embedding generation
• natural language conditioning
• mixture density network
• controllable speech synthesis