OmniCam: Unified Multimodal Video Generation via Camera Control

2025-04-03

AI Summary

Existing camera control methods suffer from cumbersome interaction, coarse-grained control, and weak multimodal coordination, resulting in videos with poor spatiotemporal consistency and inaccurate camera trajectories. To address these limitations, this paper introduces the first unified multimodal camera control framework, enabling flexible, compositional conditioning via text/video-guided trajectories jointly with image/video content references. We construct OmniTr—the first high-quality, long-sequence dataset of temporally aligned trajectory-video-caption triplets—and propose a novel architecture that synergistically integrates large language models with video diffusion models, incorporating an explicit spatiotemporal consistency modeling module. The entire framework is trained end-to-end on OmniTr. Extensive experiments demonstrate state-of-the-art performance across multiple quantitative and qualitative metrics, significantly improving spatiotemporal coherence, trajectory tracking fidelity, and visual expressiveness of generated videos.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Camera control, which achieves diverse visual effects by changing camera position and pose, has attracted widespread attention. However, existing methods face challenges such as complex interaction and limited control capabilities. To address these issues, we present OmniCam, a unified multimodal camera control framework. Leveraging large language models and video diffusion models, OmniCam generates spatio-temporally consistent videos. It supports various combinations of input modalities: the user can provide text or video with expected trajectory as camera path guidance, and image or video as content reference, enabling precise control over camera motion. To facilitate the training of OmniCam, we introduce the OmniTr dataset, which contains a large collection of high-quality long-sequence trajectories, videos, and corresponding descriptions. Experimental results demonstrate that our model achieves state-of-the-art performance in high-quality camera-controlled video generation across various metrics.

Problem

• Unified multimodal camera control for video generation
• Overcoming complex interaction and limited control in camera motion
• Generating spatio-temporally consistent videos with diverse inputs

Innovation

• Unified multimodal camera control framework
• Leverages LLMs and video diffusion models
• Supports text or video trajectory guidance