Audio ControlNet for Fine-Grained Audio Generation and Editing

2026-02-04

AI Summary

Existing text-to-audio generation models struggle to achieve fine-grained control over attributes such as loudness, pitch, and sound events. This work proposes T2A-ControlNet and its lightweight variant, T2A-Adapter—adding only 38 million parameters—which leverage a ControlNet-based conditional mechanism and an event roll representation to enable precise manipulation of multiple audio attributes atop a pretrained text-to-audio model. Notably, the approach is the first to support insertion and deletion of sound events at specified time positions. Evaluated on the AudioSet-Strong dataset, the method achieves state-of-the-art performance in both event-level and segment-level F1 scores and further demonstrates successful generalization to instruction-based controllable audio editing tasks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation
• Computer Vision: Diffusion Models for Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: User modeling for targeted and personalized online advertising

Abstract

We study the fine-grained text-to-audio (T2A) generation task. While recent models can synthesize high-quality audio from text descriptions, they often lack precise control over attributes such as loudness, pitch, and sound events. Unlike prior approaches that retrain models for specific control types, we propose to train ControlNet models on top of pre-trained T2A backbones to achieve controllable generation over loudness, pitch, and event roll. We introduce two designs, T2A-ControlNet and T2A-Adapter, and show that the T2A-Adapter model offers a more efficient structure with strong control ability. With only 38M additional parameters, T2A-Adapter achieves state-of-the-art performance on the AudioSet-Strong in both event-level and segment-level F1 scores. We further extend this framework to audio editing, proposing T2A-Editor for removing and inserting audio events at time locations specified by instructions. Models, code, dataset pipelines, and benchmarks will be released to support future research on controllable audio generation and editing.

Problem

• text-to-audio generation
• fine-grained control
• audio attributes
• sound events
• controllable audio generation

Innovation

• ControlNet
• text-to-audio generation
• audio editing
• fine-grained control
• T2A-Adapter