Efficient Audio-Visual Generation via Synchrony-Aware Cross-Modal Sparse Attention

2026-08-16

AI Summary

This study addresses the high inference costs of audio-visual generation models and the tendency of existing acceleration methods to disrupt audio-visual synchronization. We propose a synchronization-aware acceleration framework that explicitly models cross-modal interaction dynamics. By designing a protective sparse attention mechanism, this approach ensures high-fidelity computation for critical synchronization tokens while effectively sparsifying redundant interactions. Experimental results demonstrate that our method significantly enhances inference efficiency while fully preserving video and audio generation quality alongside robust audio-visual alignment. Consequently, this work achieves a dual optimization of generative efficiency and multimodal synchrony, offering a viable solution for accelerating complex audio-visual synthesis without compromising perceptual coherence or temporal alignment between modalities.

Abstract

Recent audio-visual generation models can synthesize synchronized video and sound in a unified diffusion process, but their inference cost remains high because long video token sequences require repeated attention computation across denoising steps.A variety of acceleration techniques have been developed for video generation models, including low-bit quantization, attention sparsification, and feature caching.However, since these methods are originally designed for video generation, directly applying them to audio-visual models overlooks the interactions between the audio and video branches and may therefore disrupt audio-video synchronization.We present a synchronization-aware acceleration framework for efficient audio-visual generation.Our key observation is that bidirectional audio-video cross-attention reveals structured interactions between the two branches, with high responses often concentrated on a few sound-related visual and temporal regions.Guided by this interaction pattern, we introduce a protected sparse attention strategy that preserves high-fidelity computation for synchronization-critical tokens while sparsifying redundant attention interactions.By explicitly accounting for cross-modal dependence during acceleration, our method improves inference efficiency while keeping video quality, audio quality, and audio-video synchronization.

Problem

• Audio-Visual Generation
• Inference Efficiency
• Audio-Video Synchronization
• Cross-Modal Interaction

Innovation

• Synchrony-Aware Acceleration
• Protected Sparse Attention
• Cross-Modal Interaction
• Audio-Visual Generation
• Inference Efficiency