Efficient Text-to-Audio Generation via Pruning

2026-07-14

AI Summary

This work addresses the challenge of deploying diffusion-based text-to-audio generation models, such as AudioLDM, whose U-Net backbones incur substantial computational costs. To mitigate this, the study introduces structured pruning to this domain for the first time. By analyzing parameter redundancy within U-Net convolutional blocks, the authors employ a norm-based filter pruning strategy followed by lightweight fine-tuning to recover generative performance. The proposed approach reduces U-Net parameters by 83% and multiply-accumulate operations by 39%, while preserving or even slightly improving overall audio generation quality. Crucially, the model’s ability to generate key sound events is effectively restored through fine-tuning, significantly enhancing inference efficiency and practical applicability without compromising output fidelity.

Abstract

Diffusion-based text-to-audio generative models such as AudioLDM achieve high perceptual quality and strong semantic consistency; however, their practical deployment is hindered by the substantial computational cost of the U-Net denoising backbone. In this work, we apply model pruning to improve the computational efficiency of AudioLDM, a U-Net-based text-conditioned audio latent diffusion model. We analyse parameter redundancy across U-Net convolutional blocks and evaluate a filter-pruning strategy. Pruning is guided by norm-based criteria and followed by lightweight finetuning to recover performance losses. Experimental results demonstrate that up to 83% of the parameters and 39% of the multiply-accumulate operations of U-Net have been reduced while maintaining, and in some cases improving, generation quality compared to the baseline unpruned network. We find that pruning affects AudioLDM's ability to generate certain sound events including safety-critical sounds such as gunshots, sirens, and explosions, as well as mechanical sounds such as drills and sewing machines, and other sounds such as sprays and tick-tocks, which are mostly recovered by lightweight finetuning of the pruned model.

Problem

• text-to-audio generation
• diffusion models
• computational efficiency
• U-Net
• model pruning

Innovation

• model pruning
• text-to-audio generation
• diffusion models
• U-Net acceleration
• efficient audio synthesis