Native Audio-Visual Alignment for Generation

2026-05-28

AI Summary

Existing audio-visual joint generation methods struggle to simultaneously achieve fine-grained audio-visual co-evolution and tight coupling between semantic coherence and low-level synchronization. To address this challenge, this work proposes the NAVA framework, which leverages a context-conditioned native audio-visual alignment mechanism to establish cross-modal correspondences within a dedicated interaction space and guide the joint denoising process. Key innovations include an Align-then-Fuse MMDiT architecture that enables a smooth transition from modality-aware alignment to shared denoising, and a Timbre-in-Context Conditioning mechanism that supports controllable voice timbre generation. Experimental results demonstrate that, with only 6.3B parameters, NAVA significantly improves video quality, audio-visual synchronization accuracy, audio fidelity, and timbre controllability on Verse-Bench and Seed-TTS benchmarks.

Abstract

Joint audio-video generation aims to synthesize temporally synchronized and semantically coherent visual-acoustic content. However, existing open-source methods mainly rely on either dual-tower designs with posterior alignment or fully unified tri-modal designs that mix textual context, audio and video in one shared space. The former weakens fine-grained audio-video co-evolution, while the latter couples semantic conditioning with low-level synchronization. To address these limitations, we propose NAVA, a Native Audio-Visual Alignment framework for joint audio-video generation. NAVA is built upon context-conditioned native audio-visual alignment: it first establishes audio-video correspondence in a dedicated interaction space, and then uses external context to condition the joint denoising process. Specifically, NAVA is instantiated with an Align-then-Fuse MMDiT architecture, which transitions from modality-aware audio-video alignment to modality-shared joint denoising. Furthermore, we introduce Timbre-in-Context Conditioning to associate reference timbre cues with corresponding speech spans to achieve controllable speech timbre. Experiments on Verse-Bench and Seed-TTS, together with a user study, demonstrate that NAVA achieves superior video quality, precise audio-visual synchronization, competitive audio quality, and stronger reference-timbre controllability using only 6.3B parameters.

Problem

• audio-visual alignment
• joint generation
• temporal synchronization
• semantic coherence
• timbre controllability

Innovation

• Native Audio-Visual Alignment
• Align-then-Fuse MMDiT
• Timbre-in-Context Conditioning
• Joint Audio-Video Generation
• Modality Interaction