Eigenvoice Synthesis based on Model Editing for Speaker Generation

2025-07-04

AI Summary

This work addresses zero-shot voice synthesis for unseen speakers without reference speech. We propose constructing a low-dimensional, interpretable speaker embedding space directly within the parameter space of deep neural networks—bypassing conventional reliance on acoustic features or reference audio. For the first time, we adapt the eigenvoice synthesis paradigm to DNN model editing: applying principal component analysis to the parameter space reveals a strongly semantically aligned “gender-dominant axis,” enabling fine-grained control over speaker attributes. Our method operates by sampling and editing directly in the model parameter space, yielding new speaker-specific models in a zero-shot manner. Experiments demonstrate that synthesized speech exhibits high diversity and naturalness; moreover, the parameter space manifests a clear, interpretable semantic structure. This establishes a novel paradigm for controllable voice synthesis grounded in model-parameter semantics rather than input-feature representations.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Speech
• Computer Vision: Diffusion Models for Vision

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Speaker generation task aims to create unseen speaker voice without reference speech. The key to the task is defining a speaker space that represents diverse speakers to determine the generated speaker trait. However, the effective way to define this speaker space remains unclear. Eigenvoice synthesis is one of the promising approaches in the traditional parametric synthesis framework, such as HMM-based methods, which define a low-dimensional speaker space using pre-stored speaker features. This study proposes a novel DNN-based eigenvoice synthesis method via model editing. Unlike prior methods, our method defines a speaker space in the DNN model parameter space. By directly sampling new DNN model parameters in this space, we can create diverse speaker voices. Experimental results showed the capability of our method to generate diverse speakers' speech. Moreover, we discovered a gender-dominant axis in the created speaker space, indicating the potential to control speaker attributes.

Problem

• Define speaker space for unseen voice generation
• Develop DNN-based eigenvoice synthesis via model editing
• Control speaker attributes using gender-dominant axis

Innovation

• DNN-based eigenvoice synthesis via model editing
• Defines speaker space in DNN parameter space
• Directly samples new DNN parameters for diverse voices