VIDMP3: Video Editing by Representing Motion with Pose and Position Priors

2025-10-13

AI Summary

Existing diffusion-based video editing methods suffer from temporal inconsistency, subject identity drift, and heavy reliance on manual intervention—particularly in structure-varying tasks—hindering simultaneous preservation of motion fidelity and flexible semantic/structural editing. This paper proposes a motion-preserving video editing framework that explicitly models pose and spatial position priors to construct a general, disentangled motion representation, enabling the first diffusion-based, fully automatic structure-varying editing without user guidance. Our approach integrates pose estimation, optical flow-guided feature transfer, and position-aware attention to accurately extract and propagate spatiotemporal motion characteristics from the source video. Extensive evaluations across multiple benchmarks demonstrate significant improvements over state-of-the-art methods, both qualitatively (e.g., temporal coherence) and quantitatively (e.g., lower FVD and LPIPS scores). The implementation is publicly available.

Technology Category

• Computer Vision: Motion & Tracking
• Machine Learning: Learning with Manifolds
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Motion-preserved video editing is crucial for creators, particularly in scenarios that demand flexibility in both the structure and semantics of swapped objects. Despite its potential, this area remains underexplored. Existing diffusion-based editing methods excel in structure-preserving tasks, using dense guidance signals to ensure content integrity. While some recent methods attempt to address structure-variable editing, they often suffer from issues such as temporal inconsistency, subject identity drift, and the need for human intervention. To address these challenges, we introduce VidMP3, a novel approach that leverages pose and position priors to learn a generalized motion representation from source videos. Our method enables the generation of new videos that maintain the original motion while allowing for structural and semantic flexibility. Both qualitative and quantitative evaluations demonstrate the superiority of our approach over existing methods. The code will be made publicly available at https://github.com/sandeep-sm/VidMP3.

Problem

• Addresses motion inconsistency in structure-variable video editing
• Solves subject identity drift during semantic object swapping
• Eliminates human intervention needs in motion-preserved video generation

Innovation

• Uses pose and position priors for motion representation
• Learns generalized motion from source video data
• Maintains original motion with structural flexibility