3D Scene-Adaptive Trajectory-Controllable Human Image Animation with Camera Movement

2026-06-29

AI Summary

Existing methods struggle to jointly control human motion and camera viewpoint in natural scenes for high-quality animated video generation. This work proposes a scene-adaptive human image animation framework that co-optimizes body motion and camera trajectory within a reconstructed 3D environment. The core innovations include a ground-aware 3D motion retargeting module and a viewpoint-adaptive latent fusion mechanism, which integrate point cloud geometry priors, scene visibility masks, and diffusion- or flow-based video generation models to enable user-friendly trajectory control and geometry-aware viewpoint guidance. Experiments demonstrate that the proposed approach significantly outperforms state-of-the-art methods on two standard benchmarks, achieving notable improvements across multiple video generation metrics.

Abstract

Human image animation, which aims to generate a video of a reference subject following a provided action sequence, has received increasing research interest. With the development of diffusion-based/flow-based video foundation models, existing animation works have began to upgrade the guidance information from 2D skeleton/pose to 3D modeling conditions. Despite achieving reasonable results, these approaches face challenges in synthesizing trajectory-controllable human motion within natural scene under changed camera views. In this work, we present a scene-adaptive human image animation framework that controls both human motion and camera trajectories within a reconstructed 3D environment for video generation. To achieve this, we first develop a ground-adaptive 3D motion retargeting approach to enable user-friendly motion trajectory control adapting to the changes of elevations of ground and orientations automatically. Then we design a viewpoint-adaptive latent fusion mechanism to inject point-cloud geometric priors through scene-visibility masking into the generative process, providing precise guidance of viewpoint changes under camera control. Experiments on two standard human image animation benchmark datasets demonstrate remarkable improvements of our method over the state of the arts in related video generation metics. Project page: https://robinhood256100.github.io/web-disp

Problem

• human image animation
• trajectory control
• camera movement
• 3D scene adaptation
• video generation

Innovation

• 3D motion retargeting
• camera trajectory control
• scene-adaptive animation
• latent fusion
• point-cloud priors