AnCoGen: Analysis, Control and Generation of Speech with a Masked Autoencoder

2025-01-09

AI Summary

This work addresses the challenge of unifying speech analysis, controllable synthesis, and generative modeling within a single framework. We propose AnCoGen—a unified, isomorphic architecture based on masked autoencoding—that jointly models speech analysis (e.g., speaker identity, pitch, linguistic content, loudness estimation), attribute-disentangled control, and high-fidelity waveform generation for the first time. Leveraging multi-task joint training and attribute-conditional generation, AnCoGen enables fine-grained, multi-dimensional semantic editing of speech attributes. Extensive experiments demonstrate state-of-the-art performance on voice re-synthesis, pitch estimation and modification, and speech enhancement—validating its strong cross-task generalization and precise controllability. AnCoGen establishes a novel paradigm for speech representation learning and controllable generation, bridging traditionally disjoint objectives in speech processing.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Deep Generative Models & Autoencoders
• Cognitive Modeling & Cognitive Systems: Affective Computing

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Agentic search
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

This article introduces AnCoGen, a novel method that leverages a masked autoencoder to unify the analysis, control, and generation of speech signals within a single model. AnCoGen can analyze speech by estimating key attributes, such as speaker identity, pitch, content, loudness, signal-to-noise ratio, and clarity index. In addition, it can generate speech from these attributes and allow precise control of the synthesized speech by modifying them. Extensive experiments demonstrated the effectiveness of AnCoGen across speech analysis-resynthesis, pitch estimation, pitch modification, and speech enhancement.

Problem

• Speech Analysis
• Speech Synthesis
• Naturalness Control

Innovation

• SpeechAnalysis
• IntegratedModel
• SpeechGeneration