UniSAE: Unified Speech Attribute Editing on Speaker, Emotion and Low-Level Content via Discrete Phonetic Posteriorgram Modelling

2026-06-30

AI Summary

Existing speech editing methods are largely confined to word-level content modification and treat speaker identity, emotion, and linguistic content as separate tasks, lacking fine-grained control and unified editing capabilities. This work proposes UniSAE, a unified framework that, for the first time, enables composable editing of speaker characteristics, emotional attributes, and multi-granular speech content within a single architecture. UniSAE introduces Discrete Phoneme Posterior Graphs (DPPGs) as sub-phonemic representations, integrating an autoregressive content transformer with a diffusion-based acoustic decoder, alongside disentangled speaker and emotion embeddings. The framework supports high-precision joint editing ranging from sub-phonemic to word-level units, significantly enhancing editing flexibility and control accuracy while preserving naturalness in the synthesized speech.

Abstract

Speech editing aims to modify specific portions of an utterance while preserving the remaining speech. Existing approaches primarily focus on word-level content modification and typically treat content, speaker, and emotion editing as separate tasks, limiting both editing granularity and flexibility. We propose UniSAE, a unified speech attribute editing framework which supports composable speaker, emotion and content editing from sub-phoneme to word level within a single architecture. UniSAE introduces a Discrete Phonetic PosteriorGram (DPPG) representation that factorizes speech content into discrete tokens encoding phoneme identity, pronunciation variants, and duration, enabling direct phoneme- and sub-phoneme-level editing. For higher-level modifications, an autoregressive content transformer predicts edited DPPG sequences for word-level content editing. The edited sequences are rendered into speech by a diffusion-based acoustic decoder, conditioned on disentangled speaker and emotion representations. Experimental results demonstrate that the proposed unified framework supports precise speaker and emotion control, content editing at multiple granularities, and joint modification of all three attributes within a single framework.

Problem

• speech editing
• speaker editing
• emotion editing
• content editing
• editing granularity

Innovation

• Discrete Phonetic PosteriorGram
• unified speech editing
• sub-phoneme editing
• diffusion-based acoustic decoder
• disentangled speaker and emotion representation