Virtually Being: Customizing Camera-Controllable Video Diffusion Models with Multi-View Performance Captures

2025-10-15

AI Summary

This work addresses three key challenges in virtual production using video diffusion models: preserving multi-view identity consistency, enabling 3D camera controllability, and supporting multi-subject customization. To this end, we propose a joint optimization framework that (1) leverages 4D Gaussian Splatting (4DGS) for voxelized performance relighting, augmented by video relighting-based data enhancement to diversify illumination conditions; (2) incorporates a noise fusion mechanism for efficient, inference-time multi-character composition; and (3) jointly fine-tunes open-source video diffusion models to enhance illumination adaptability and spatial layout control. Experiments demonstrate significant improvements over state-of-the-art methods in multi-view identity preservation, camera trajectory fidelity, and illumination robustness. The method also achieves superior video quality and personalized subject accuracy, establishing a new paradigm for high-fidelity, controllable video generation in virtual production.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Systems and Infrastructure for Web, Mobile and WoT: Virtualization and resource management in Web systems and infrastructures
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We introduce a framework that enables both multi-view character consistency and 3D camera control in video diffusion models through a novel customization data pipeline. We train the character consistency component with recorded volumetric capture performances re-rendered with diverse camera trajectories via 4D Gaussian Splatting (4DGS), lighting variability obtained with a video relighting model. We fine-tune state-of-the-art open-source video diffusion models on this data to provide strong multi-view identity preservation, precise camera control, and lighting adaptability. Our framework also supports core capabilities for virtual production, including multi-subject generation using two approaches: joint training and noise blending, the latter enabling efficient composition of independently customized models at inference time; it also achieves scene and real-life video customization as well as control over motion and spatial layout during customization. Extensive experiments show improved video quality, higher personalization accuracy, and enhanced camera control and lighting adaptability, advancing the integration of video generation into virtual production. Our project page is available at: https://eyeline-labs.github.io/Virtually-Being.

Problem

• Achieving multi-view character consistency in video generation
• Enabling precise 3D camera control in video diffusion models
• Supporting lighting adaptability and multi-subject virtual production

Innovation

• Multi-view character consistency via 4D Gaussian Splatting
• Camera control and lighting adaptability in video diffusion
• Multi-subject generation through joint training and noise blending