V2M4: 4D Mesh Animation Reconstruction from a Single Monocular Video

2025-03-11

AI Summary

This work introduces the first monocular video 4D animation reconstruction method built upon a native 3D mesh generation model, addressing key challenges in 4D reconstruction—including pose misalignment, geometry-texture misregistration, and inter-frame topological inconsistency. The method establishes a joint optimization framework: (i) differentiable camera parameter search for accurate per-frame pose estimation; (ii) implicit conditional embedding to enhance temporal coherence; and (iii) integrated ICP-based geometric registration with global UV-space texture optimization to ensure cross-frame consistency in both mesh geometry and appearance. Evaluated on complex dynamic scenes—such as dance and sports—the approach achieves high-fidelity, topologically consistent, and temporally smooth 4D mesh animations. The output is compatible with industrial-standard formats (e.g., USD, glTF), enabling plug-and-play integration into mainstream graphics and game engines.

Technology Category

• Computer Vision: 3D Computer Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Learning with Manifolds

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

We present V2M4, a novel 4D reconstruction method that directly generates a usable 4D mesh animation asset from a single monocular video. Unlike existing approaches that rely on priors from multi-view image and video generation models, our method is based on native 3D mesh generation models. Naively applying 3D mesh generation models to generate a mesh for each frame in a 4D task can lead to issues such as incorrect mesh poses, misalignment of mesh appearance, and inconsistencies in mesh geometry and texture maps. To address these problems, we propose a structured workflow that includes camera search and mesh reposing, condition embedding optimization for mesh appearance refinement, pairwise mesh registration for topology consistency, and global texture map optimization for texture consistency. Our method outputs high-quality 4D animated assets that are compatible with mainstream graphics and game software. Experimental results across a variety of animation types and motion amplitudes demonstrate the generalization and effectiveness of our method. Project page:https://windvchen.github.io/V2M4/.

Problem

• Generates 4D mesh animation from single video
• Addresses mesh pose and appearance misalignment
• Ensures topology and texture consistency in 4D

Innovation

• Native 3D mesh generation models for 4D reconstruction
• Structured workflow for mesh and texture consistency
• High-quality 4D animation compatible with mainstream software