SwiftAudio: Data-Efficient Caption-Only Distillation for One-Step Text-to-Audio Diffusion-based Generation

2026-06-30

AI Summary

Existing single-step text-to-audio generation methods still rely on paired data for distillation, struggling to balance efficiency and quality. This work proposes SwiftAudio, the first approach to introduce Variational Score Distillation (VSD) into audio generation, enabling training using only textual descriptions without any ground-truth audio samples. By incorporating a temporal smoothing regularization objective, SwiftAudio achieves high-quality single-step audio synthesis with merely approximately 45K text descriptions. It attains state-of-the-art performance among unpaired single-step methods on both AudioCaps and Clotho benchmarks, significantly narrowing the quality gap with multi-step diffusion models.

Abstract

Diffusion-based text-to-audio (TTA) models achieve impressive synthesis quality but suffer from high inference latency due to iterative multi-step denoising. Existing one-step approaches alleviate this issue but still rely on paired text--audio data during distillation. To address these limitations, we propose SwiftAudio, a one-step TTA framework that performs audio-free distillation from a pretrained diffusion teacher using only text captions. Specifically, we adapt Variational Score Distillation (VSD) to the audio domain and introduce a temporal smoothness regularization objective to encourage coherent latent audio representations. This design enables the student model to inherit the teacher's generative prior without requiring paired audio supervision and allows effective training with only approximately 45K captions. Experiments on AudioCaps and Clotho demonstrate that SwiftAudio achieves state-of-the-art performance among strict one-step methods and substantially narrows the gap to multi-step diffusion systems. Project page: https://swiftaudio.org/

Problem

• text-to-audio
• diffusion models
• one-step generation
• data-efficient distillation
• audio-free distillation

Innovation

• one-step generation
• audio-free distillation
• Variational Score Distillation
• text-to-audio
• temporal smoothness regularization