UnityVideo: Unified Multi-Modal Multi-Task Learning for Enhancing World-Aware Video Generation

2025-12-08

AI Summary

Current video generation models are constrained by unimodal conditional inputs, resulting in weak cross-modal interaction, insufficient modality diversity, and consequently limited world modeling capability and poor physical consistency. To address these limitations, we propose a multimodal world-aware video generation framework. Our method introduces a dynamic noise-scheduling mechanism and a context-aware modality switcher to unify training across heterogeneous inputs—including segmentation masks, skeletal poses, DensePose, optical flow, and depth maps. We curate a large-scale multimodal video dataset comprising 1.3 million samples. Leveraging diffusion models, our architecture integrates modular parameterization, multi-task loss optimization, and context learning. Experiments demonstrate substantial improvements over state-of-the-art methods across multiple benchmarks in terms of generation quality, spatiotemporal coherence, and zero-shot transferability, with outputs exhibiting stronger adherence to real-world physical principles.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Recent video generation models demonstrate impressive synthesis capabilities but remain limited by single-modality conditioning, constraining their holistic world understanding. This stems from insufficient cross-modal interaction and limited modal diversity for comprehensive world knowledge representation. To address these limitations, we introduce UnityVideo, a unified framework for world-aware video generation that jointly learns across multiple modalities (segmentation masks, human skeletons, DensePose, optical flow, and depth maps) and training paradigms. Our approach features two core components: (1) dynamic noising to unify heterogeneous training paradigms, and (2) a modality switcher with an in-context learner that enables unified processing via modular parameters and contextual learning. We contribute a large-scale unified dataset with 1.3M samples. Through joint optimization, UnityVideo accelerates convergence and significantly enhances zero-shot generalization to unseen data. We demonstrate that UnityVideo achieves superior video quality, consistency, and improved alignment with physical world constraints. Code and data can be found at: https://github.com/dvlab-research/UnityVideo

Problem

• Enhances world-aware video generation via multi-modal learning
• Unifies heterogeneous training paradigms and multiple modalities
• Improves video quality, consistency, and physical world alignment

Innovation

• Unified multi-modal learning across diverse visual modalities
• Dynamic noising and modality switcher with in-context learner
• Joint optimization for enhanced generalization and world-aware generation