MagicPose4D: Crafting Articulated Models with Appearance and Motion Control

2024-05-22

arXiv.org

AI Summary

Existing 4D content generation methods rely on text prompts, limiting precise control over complex or rare motions. MagicPose4D addresses this by introducing a novel two-stage 4D generation framework that accepts monocular videos or mesh sequences as motion priors, enabling fine-grained co-modeling of appearance and motion. Its key contributions are: (1) a cross-category motion transfer module; (2) a global-local Chamfer loss combined with kinematic-chain-based skeletal constraints to ensure both geometric fidelity and physical plausibility; and (3) a multi-source supervision strategy integrating 2D image reconstruction, pseudo-3D supervision, dynamic rigid interpolation, and skeleton-driven motion transfer. Experiments demonstrate significant improvements over state-of-the-art methods in motion accuracy, temporal coherence, and cross-category generalization. Notably, MagicPose4D achieves robust motion transfer across categories without fine-tuning.

Technology Category

• Computer Vision: Motion & Tracking
• Machine Learning: Multimodal Learning
• Intelligent Robots: Motion and Path Planning

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

With the success of 2D and 3D visual generative models, there is growing interest in generating 4D content. Existing methods primarily rely on text prompts to produce 4D content, but they often fall short of accurately defining complex or rare motions. To address this limitation, we propose MagicPose4D, a novel framework for refined control over both appearance and motion in 4D generation. Unlike current 4D generation methods, MagicPose4D accepts monocular videos or mesh sequences as motion prompts, enabling precise and customizable motion control. MagicPose4D comprises two key modules: (i) Dual-Phase 4D Reconstruction Module, which operates in two phases. The first phase focuses on capturing the model's shape using accurate 2D supervision and less accurate but geometrically informative 3D pseudo-supervision without imposing skeleton constraints. The second phase extracts the 3D motion (skeleton poses) using more accurate pseudo-3D supervision, obtained in the first phase and introduces kinematic chain-based skeleton constraints to ensure physical plausibility. Additionally, we propose a Global-local Chamfer loss that aligns the overall distribution of predicted mesh vertices with the supervision while maintaining part-level alignment without extra annotations. (ii) Cross-category Motion Transfer Module, which leverages the extracted motion from the 4D reconstruction module and uses a kinematic-chain-based skeleton to achieve cross-category motion transfer. It ensures smooth transitions between frames through dynamic rigidity, facilitating robust generalization without additional training. Through extensive experiments, we demonstrate that MagicPose4D significantly improves the accuracy and consistency of 4D content generation, outperforming existing methods in various benchmarks.

Problem

• Enables precise motion control in 4D generation using video or mesh inputs
• Improves 4D reconstruction via dual-phase shape and motion extraction
• Facilitates cross-category motion transfer with kinematic-chain-based skeleton

Innovation

• Uses monocular videos for motion control
• Dual-Phase 4D Reconstruction for shape and motion
• Cross-category Motion Transfer with kinematic chains