A Review on Score-based Generative Models for Audio Applications

2025-06-10

AI Summary

Prior research on audio diffusion models lacks systematic analysis of design choices and task-oriented design principles. Method: Framed within a unified score modeling paradigm, this work pioneers the integration of diffusion models with emerging generative paradigms—including flow matching—and establishes principled guidelines for enhancing audio fidelity and enabling fine-grained conditional control (e.g., text or noisy speech). Contribution/Results: We release AudioDiffuser, a modular, extensible PyTorch codebase, and conduct standardized benchmarking across three core tasks—audio generation, speech enhancement, and text-to-speech—with fully reproducible implementations. Our work advances theoretical unification of audio generative models, promotes engineering standardization, and facilitates practical deployment.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Diffusion models have emerged as powerful deep generative techniques, producing high-quality and diverse samples in applications in various domains including audio. These models have many different design choices suitable for different applications, however, existing reviews lack in-depth discussions of these design choices. The audio diffusion model literature also lacks principled guidance for the implementation of these design choices and their comparisons for different applications. This survey provides a comprehensive review of diffusion model design with an emphasis on design principles for quality improvement and conditioning for audio applications. We adopt the score modeling perspective as a unifying framework that accommodates various interpretations, including recent approaches like flow matching. We systematically examine the training and sampling procedures of diffusion models, and audio applications through different conditioning mechanisms. To address the lack of audio diffusion model codebases and to promote reproducible research and rapid prototyping, we introduce an open-source codebase at https://github.com/gzhu06/AudioDiffuser that implements our reviewed framework for various audio applications. We demonstrate its capabilities through three case studies: audio generation, speech enhancement, and text-to-speech synthesis, with benchmark evaluations on standard datasets.

Problem

• Lack of in-depth discussion on diffusion model design choices
• Absence of guidance for implementing and comparing design choices
• Need for open-source codebase for audio diffusion model research

Innovation

• Comprehensive review of diffusion model design
• Score modeling as unifying framework
• Open-source codebase for audio applications