UniVerse-1: Unified Audio-Video Generation via Stitching of Experts

2025-09-07

AI Summary

This work addresses the challenge of precise temporal and semantic alignment among environmental sounds, speech, and visual frames in audio-visual synchronous generation. We propose the Stitching of Experts (SoE) framework: it composes pre-trained video and audio foundation models—avoiding costly end-to-end training—and introduces an online audio-visual annotation pipeline to achieve frame-level synchronization, thereby eliminating temporal drift inherent in text-based supervision. Fine-tuned on approximately 7,600 hours of audio-visual data, SoE significantly outperforms baseline methods on environmental sound generation and speech–motion synchronization tasks, achieving performance competitive with Veo3 on our newly constructed benchmark, Verse-Bench. Our key contribution is the first integration of expert model composition with online audio-visual annotation, establishing a novel paradigm for efficient, high-fidelity audio-visual co-generation.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Computer Vision: Language and Vision
• Natural Language Processing: Generation

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

We introduce UniVerse-1, a unified, Veo-3-like model capable of simultaneously generating coordinated audio and video. To enhance training efficiency, we bypass training from scratch and instead employ a stitching of experts (SoE) technique. This approach deeply fuses the corresponding blocks of pre-trained video and music generation experts models, thereby fully leveraging their foundational capabilities. To ensure accurate annotations and temporal alignment for both ambient sounds and speech with video content, we developed an online annotation pipeline that processes the required training data and generates labels during training process. This strategy circumvents the performance degradation often caused by misalignment text-based annotations. Through the synergy of these techniques, our model, after being finetuned on approximately 7,600 hours of audio-video data, produces results with well-coordinated audio-visuals for ambient sounds generation and strong alignment for speech generation. To systematically evaluate our proposed method, we introduce Verse-Bench, a new benchmark dataset. In an effort to advance research in audio-video generation and to close the performance gap with state-of-the-art models such as Veo3, we make our model and code publicly available. We hope this contribution will benefit the broader research community. Project page: https://dorniwang.github.io/UniVerse-1/.

Problem

• Simultaneously generating coordinated audio and video content
• Overcoming performance degradation from misaligned text annotations
• Closing performance gap with state-of-the-art audio-video generation models

Innovation

• Stitching pre-trained video and music experts
• Online annotation pipeline for temporal alignment
• Fine-tuned on 7600 hours audio-video data