AVTok: 1D Unified Tokenization for Holistic Audio-Video Generation

2026-06-29

AI Summary

Existing audio-visual generation methods struggle to achieve fine-grained synchronization and semantic alignment due to modality representation gaps and high computational overhead inherent in dual-branch architectures. This work proposes AVTok, the first unified one-dimensional tokenizer designed for holistic audio-visual generation. AVTok employs a dual-stream Transformer architecture with a shared encoder-decoder backbone and modality-specific learnable queries to compress both modalities into a unified one-dimensional latent representation. To address inter-modal information imbalance, a hierarchical progressive training strategy is introduced. AVTok demonstrates superior performance in reconstruction tasks and significantly enhances generation quality and temporal synchrony in downstream applications—including audio-to-video, video-to-audio, and class-conditioned joint generation—thereby advancing the feasibility of unified multimodal foundation models for audio-visual synthesis.

Abstract

Audio-video generation has recently gained unprecedented research attention, aiming to synthesize high-quality sounding video content with fine-grained synchronization and semantic alignment between the auditory and visual components. The preceding methods predominantly adopt a dual-branch design with separate tokenization and generation modules per modality, neglecting the representation gap while necessitating intensive computational resources for proper training. Inspired by recent advancements in one-dimensional visual tokenization, we present \textbf{AVTok}, a novel unified tokenizer designated for holistic audio-video generation. AVTok features a dual-stream transformer-based architecture with shared encoder-decoder and modal-specific learnable queries to efficiently and effectively encode an audio-video pair into a compact one-dimensional latent representation with a unified codebook. To cope with the heterogeneous information imbalance that hinders AVTok from exploiting aligned audio-visual information, we devise a hierarchical training strategy to progressively realize reconstruction capabilities for each modality. Extensive experiments demonstrate that AVTok excels both in audio-video reconstruction and when integrated into downstream pipelines for audio-to-video, video-to-audio, and class-conditional joint audio-video generation. AVTok paves the way for the challenge of joint audio-video tokenization and provides a potential direction to build unified large multimodal models for audio-video generation.

Problem

• audio-video generation
• unified tokenization
• multimodal representation
• modality alignment
• heterogeneous information

Innovation

• unified tokenization
• audio-video generation
• dual-stream transformer
• one-dimensional latent representation
• hierarchical training