Efficient Speech Enhancement via Embeddings from Pre-trained Generative Audioencoders

2025-06-13

AI Summary

This work addresses speech enhancement under noisy conditions. We propose an efficient end-to-end framework that leverages embeddings from a pretrained generative audio encoder (e.g., AudioLDM encoder): noisy speech is first mapped to a deep generative embedding space; a lightweight Transformer encoder then performs denoising within this embedding space; finally, a neural vocoder (e.g., DAC or EnCodec) reconstructs clean speech. Our key contribution is the first integration of generative—rather than conventional discriminative—pretrained audio encoder embeddings into speech enhancement, enabling joint optimization of embedding representation and waveform reconstruction. The modular, decoupled architecture supports flexible substitution of individual components. Experiments demonstrate state-of-the-art performance in speech quality (MOS), intelligibility, and speaker fidelity, while reducing model parameters by 47% compared to prior approaches.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation
• Computer Vision: Diffusion Models for Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Recent research has delved into speech enhancement (SE) approaches that leverage audio embeddings from pre-trained models, diverging from time-frequency masking or signal prediction techniques. This paper introduces an efficient and extensible SE method. Our approach involves initially extracting audio embeddings from noisy speech using a pre-trained audioencoder, which are then denoised by a compact encoder network. Subsequently, a vocoder synthesizes the clean speech from denoised embeddings. An ablation study substantiates the parameter efficiency of the denoise encoder with a pre-trained audioencoder and vocoder. Experimental results on both speech enhancement and speaker fidelity demonstrate that our generative audioencoder-based SE system outperforms models utilizing discriminative audioencoders. Furthermore, subjective listening tests validate that our proposed system surpasses an existing state-of-the-art SE model in terms of perceptual quality.

Problem

• Improves speech enhancement using pre-trained audio embeddings
• Denoises speech efficiently with compact encoder network
• Outperforms discriminative models in perceptual quality

Innovation

• Pre-trained audioencoder extracts noisy speech embeddings
• Compact encoder network denoises the embeddings
• Vocoder synthesizes clean speech from denoised embeddings