TADA! Tuning Audio Diffusion Models through Activation Steering

2026-02-12

AI Summary

This work addresses the limited understanding of how current audio diffusion models internally represent high-level musical semantics—such as instruments, vocals, style, rhythm, and emotion—and the consequent lack of precise control over generation. By applying activation patching to identify critical attention layers and combining contrastive activation addition with sparse autoencoders, we reveal—for the first time—the existence of shared yet functionally specialized subsets of attention heads that explicitly encode distinct high-level musical concepts. Leveraging this insight, we demonstrate high-precision intervention and editing of specific musical elements in generated audio, substantially enhancing both the interpretability and controllability of diffusion-based audio synthesis.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding

Abstract

Audio diffusion models can synthesize high-fidelity music from text, yet their internal mechanisms for representing high-level concepts remain poorly understood. In this work, we use activation patching to demonstrate that distinct semantic musical concepts, such as the presence of specific instruments, vocals, or genre characteristics, are controlled by a small, shared subset of attention layers in state-of-the-art audio diffusion architectures. Next, we demonstrate that applying Contrastive Activation Addition and Sparse Autoencoders in these layers enables more precise control over the generated audio, indicating a direct benefit of the specialization phenomenon. By steering activations of the identified layers, we can alter specific musical elements with high precision, such as modulating tempo or changing a track's mood.

Problem

• audio diffusion models
• semantic musical concepts
• activation steering
• high-level representation
• precise control

Innovation

• activation steering
• audio diffusion models
• activation patching
• contrastive activation addition
• sparse autoencoders