AutoSIFT: Automatic Style Sifting for Controllable Speech Generation with Arbitrary Style Infilling

2026-07-14

AI Summary

Existing text-to-speech synthesis methods struggle to precisely edit specific semantic styles—such as emotion, age, or gender—while preserving the unspoken prosodic and speaker-specific details present in a reference utterance. To address this limitation, this work proposes AutoSIFT, a novel framework that disentangles speaking style into text-describable categorical style and non-describable residual style. AutoSIFT employs a style separator to extract category-aware prototypes and an arbitrary style infuser to selectively inject residual components. This approach enables, for the first time, independent editing of arbitrary semantic style categories, overcoming the coupling constraints inherent in conventional methods. By doing so, it achieves high-fidelity, expressive speech synthesis with fine-grained, flexible style control—making it particularly suitable for professional applications such as film dubbing and game voice generation.

Abstract

State-of-the-art text-to-speech (TTS) models achieve impressive naturalness and expressiveness, yet fine-grained, disentangled control over speaking styles remains challenging. In professional scenarios such as film dubbing, game voice acting, and video content generation, users often need to modify a specific style category, such as emotion, age, or gender, while preserving all others. Existing style-controllable TTS methods typically rely on either text-described styles or speech-reference style transfer, making it difficult to jointly control explicit semantic attributes and preserve subtle, text-undescribed prosodic details. We propose AutoSIFT, a controllable speech generation framework for category-level style editing. AutoSIFT decomposes speaking style into known text-describable categories and unknown residual styles that capture non-verbal prosody and speaker-specific nuances. It consists of a generalized Style Disentangler, which extracts category-aware style prototypes from reference speech, and an Arbitrary Style Infiller, which selectively infills unspecified style categories from the reference. By replacing only text-specified style categories while preserving residual speech-derived styles, AutoSIFT enables natural, expressive, and highly customizable speech generation.

Problem

• controllable speech generation
• style disentanglement
• text-to-speech
• style editing
• prosody preservation

Innovation

• style disentanglement
• controllable speech synthesis
• arbitrary style infilling
• residual prosody preservation
• category-level style editing