Hybrid Diffusion Transformer for Instruction-Guided Audio Editing via Rectified Flow

2026-06-18

AI Summary

Existing audio editing methods face limitations in long-range semantic alignment, precise instruction grounding, and computational efficiency. This work proposes a two-stage diffusion Transformer architecture based on rectified flow matching: a low-resolution stage employs joint attention for coarse-grained semantic alignment, followed by a high-resolution stage that alternates between joint and cross-attention to refine editing details. The approach innovatively integrates a coarse-to-fine hybrid attention mechanism, substantially reducing computational complexity while surpassing the performance bottlenecks of conventional U-Nets and fully joint-attention Transformers. Experiments demonstrate significant improvements on challenging tasks involving overlapping audio events and complex editing instructions, achieving markedly higher editing efficiency through a compact model structure.

Abstract

Audio editing aims to modify specific content in an existing audio clip according to a natural language instruction while preserving the remaining acoustic content. Despite the remarkable progress of diffusion models, existing training-based editing methods mainly rely on the local inductive biases and cross-attention interaction in convolutional U-Net backbones, which often hinder long-range semantic alignment and precise understanding and localization of instructions. In contrast, diffusion transformers provide stronger global modeling and multimodal fusion, but existing editing architectures usually adopt a simple stack of MMDiT and DiT blocks. Applying joint attention over concatenated audio and text tokens in all blocks results in quadratic complexity with respect to token length. To balance editing performance and efficiency, we propose a hybrid two-stage diffusion transformer architecture for instruction-guided audio editing based on rectified flow matching. It performs joint attention over audio and text tokens to establish coarse semantic alignment at low-resolution stage, then switches to alternating joint-attention and cross-attention blocks to refine editing details at high-resolution stage. This coarse-to-fine strategy enables efficient and accurate instruction-guided audio editing. Experiments show that the proposed framework achieves notable performance gains on challenging editing tasks involving overlapping audio events and complex instructions, while substantially improving editing efficiency with a compact model.

Problem

• audio editing
• instruction-guided
• diffusion models
• semantic alignment
• computational efficiency

Innovation

• Hybrid Diffusion Transformer
• Instruction-Guided Audio Editing
• Rectified Flow Matching
• Coarse-to-Fine Attention
• Multimodal Fusion