Uni3C: Unifying Precisely 3D-Enhanced Camera and Human Motion Controls for Video Generation

2025-04-21

AI Summary

Existing video generation methods typically decouple camera motion and human pose control, but suffer from limited control accuracy and generation quality due to the scarcity of high-quality jointly annotated data. To address this, we propose the first unified 3D-augmented framework for camera–human co-control. Our approach introduces a plug-and-play PCDController module that enables motion-decoupled training via monocular depth estimation and point-cloud back-projection. Furthermore, we design a joint alignment guidance mechanism between scene point clouds and SMPL-X avatars in the global 3D world coordinate system. The framework requires no backbone modification and supports flexible adaptation—either freezing or fine-tuning the base model. Evaluated on a newly constructed challenging benchmark, our method achieves state-of-the-art performance in both camera control precision and human motion naturalness, demonstrating strong generalization capability and backbone-agnostic robustness.

Technology Category

• Computer Vision: Motion & Tracking
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Camera and human motion controls have been extensively studied for video generation, but existing approaches typically address them separately, suffering from limited data with high-quality annotations for both aspects. To overcome this, we present Uni3C, a unified 3D-enhanced framework for precise control of both camera and human motion in video generation. Uni3C includes two key contributions. First, we propose a plug-and-play control module trained with a frozen video generative backbone, PCDController, which utilizes unprojected point clouds from monocular depth to achieve accurate camera control. By leveraging the strong 3D priors of point clouds and the powerful capacities of video foundational models, PCDController shows impressive generalization, performing well regardless of whether the inference backbone is frozen or fine-tuned. This flexibility enables different modules of Uni3C to be trained in specific domains, i.e., either camera control or human motion control, reducing the dependency on jointly annotated data. Second, we propose a jointly aligned 3D world guidance for the inference phase that seamlessly integrates both scenic point clouds and SMPL-X characters to unify the control signals for camera and human motion, respectively. Extensive experiments confirm that PCDController enjoys strong robustness in driving camera motion for fine-tuned backbones of video generation. Uni3C substantially outperforms competitors in both camera controllability and human motion quality. Additionally, we collect tailored validation sets featuring challenging camera movements and human actions to validate the effectiveness of our method.

Problem

• Unifies camera and human motion controls for video generation
• Overcomes limited data with high-quality joint annotations
• Enhances precision in 3D control via point clouds

Innovation

• Plug-and-play control module with point clouds
• Jointly aligned 3D world guidance integration
• Unified framework for camera and human motion