FlashAudio: Rectified Flows for Fast and High-Fidelity Text-to-Audio Generation

2024-10-16

arXiv.org

AI Summary

To address the high sampling step count and low inference efficiency of latent diffusion models (LDMs) in text-to-audio generation, this paper proposes a single-step, high-fidelity generation framework based on rectified flows. Our method introduces three key innovations: (1) the Bifocal sampler, which optimizes temporal scheduling to enhance trajectory linearity; (2) the immiscible flow batch-pairing mechanism, which reduces inter-sample distance deviation; and (3) Anchored Optimization, a guidance strategy that mitigates error accumulation in classifier-free guidance (CFG). Experiments demonstrate that our approach achieves superior audio quality in a single denoising step—surpassing 100-step LDM baselines—while attaining a real-time factor of 400× on a single RTX 4090 GPU. This represents a substantial leap beyond the performance ceiling of existing single-step diffusion methods for audio synthesis.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

Recent advancements in latent diffusion models (LDMs) have markedly enhanced text-to-audio generation, yet their iterative sampling processes impose substantial computational demands, limiting practical deployment. While recent methods utilizing consistency-based distillation aim to achieve few-step or single-step inference, their one-step performance is constrained by curved trajectories, preventing them from surpassing traditional diffusion models. In this work, we introduce FlashAudio with rectified flows to learn straight flow for fast simulation. To alleviate the inefficient timesteps allocation and suboptimal distribution of noise, FlashAudio optimizes the time distribution of rectified flow with Bifocal Samplers and proposes immiscible flow to minimize the total distance of data-noise pairs in a batch vias assignment. Furthermore, to address the amplified accumulation error caused by the classifier-free guidance (CFG), we propose Anchored Optimization, which refines the guidance scale by anchoring it to a reference trajectory. Experimental results on text-to-audio generation demonstrate that FlashAudio's one-step generation performance surpasses the diffusion-based models with hundreds of sampling steps on audio quality and enables a sampling speed of 400x faster than real-time on a single NVIDIA 4090Ti GPU.

Problem

• Fast high-fidelity text-to-audio generation with rectified flows
• Optimizing time distribution and noise allocation for efficiency
• Reducing error from classifier-free guidance in one-step inference

Innovation

• Rectified flows enable fast simulation
• Bifocal Samplers optimize time distribution
• Anchored Optimization refines guidance scale