ActCam: Zero-Shot Joint Camera and 3D Motion Control for Video Generation

2026-05-07

AI Summary

Existing video generation methods struggle to simultaneously achieve fine-grained control over character motion and camera trajectories. This work proposes a zero-shot approach that leverages a pre-trained image-to-video diffusion model without any additional training, employing a two-stage conditional guidance strategy. By integrating temporally consistent 3D pose estimates with sparse depth maps, the method enables geometrically coherent character motion transfer and per-frame camera parameter control. It represents the first zero-shot framework capable of jointly manipulating both camera trajectory and 3D action. Extensive evaluations demonstrate significant improvements in motion fidelity and camera adherence across multiple benchmarks, with human assessments particularly highlighting its superior performance in scenarios involving large viewpoint changes.

Abstract

For artistic applications, video generation requires fine-grained control over both performance and cinematography, i.e., the actor's motion and the camera trajectory. We present ActCam, a zero-shot method for video generation that jointly transfers character motion from a driving video into a new scene and enables per-frame control of intrinsic and extrinsic camera parameters. ActCam builds on any pretrained image-to-video diffusion model that accepts conditioning in terms of scene depth and character pose. Given a source video with a moving character and a target camera motion, ActCam generates pose and depth conditions that remain geometrically consistent across frames. We then run a single sampling process with a two-phase conditioning schedule: early denoising steps condition on both pose and sparse depth to enforce scene structure, after which depth is dropped and pose-only guidance refines high-frequency details without over-constraining the generation. We evaluate ActCam on multiple benchmarks spanning diverse character motions and challenging viewpoint changes. We find that, compared to pose-only control and other pose and camera methods, ActCam improves camera adherence and motion fidelity, and is preferred in human evaluations, especially under large viewpoint changes. Our results highlight that careful camera-consistent conditioning and staged guidance can enable strong joint camera and motion control without training. Project page: https://elkhomar.github.io/actcam/.

Problem

• video generation
• camera control
• 3D motion
• zero-shot
• joint control

Innovation

• zero-shot video generation
• joint camera and motion control
• two-phase conditioning
• geometric consistency
• diffusion models