VividCam: Learning Unconventional Camera Motions from Virtual Synthetic Videos

2025-10-28

AI Summary

Current text-to-video models exhibit poor generalization to unconventional camera motions, limiting artistic video generation. To address this, we propose VividCam—a novel training paradigm that leverages lightweight, controllable synthetic video data rendered from low-poly 3D scenes in Unity, eliminating reliance on scarce real-world rare-shot footage. Methodologically, VividCam introduces a multi-level disentanglement strategy: it decouples camera motion modeling from appearance generation, explicitly suppresses synthesis artifacts, enhances robustness of motion representations, and mitigates domain shift. Built upon a diffusion-based architecture, the framework supports end-to-end training. Experiments demonstrate that, using only simple synthetic data, VividCam achieves precise and diverse control over complex camera trajectories—including spirals, dives, and orbits—in real-video generation, significantly improving cross-domain generalization.

Technology Category

• Computer Vision: Computational Photography, Image & Video Synthesis
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Although recent text-to-video generative models are getting more capable of following external camera controls, imposed by either text descriptions or camera trajectories, they still struggle to generalize to unconventional camera motions, which is crucial in creating truly original and artistic videos. The challenge lies in the difficulty of finding sufficient training videos with the intended uncommon camera motions. To address this challenge, we propose VividCam, a training paradigm that enables diffusion models to learn complex camera motions from synthetic videos, releasing the reliance on collecting realistic training videos. VividCam incorporates multiple disentanglement strategies that isolates camera motion learning from synthetic appearance artifacts, ensuring more robust motion representation and mitigating domain shift. We demonstrate that our design synthesizes a wide range of precisely controlled and complex camera motions using surprisingly simple synthetic data. Notably, this synthetic data often consists of basic geometries within a low-poly 3D scene and can be efficiently rendered by engines like Unity. Our video results can be found in https://wuqiuche.github.io/VividCamDemoPage/ .

Problem

• Learning unconventional camera motions from synthetic videos
• Overcoming limited training data for artistic video generation
• Isolating camera motion learning from synthetic appearance artifacts

Innovation

• Learning camera motions from synthetic virtual videos
• Using disentanglement strategies to isolate motion from artifacts
• Employing basic geometries in low-poly 3D scenes for rendering