Through-The-Mask: Mask-based Motion Trajectories for Image-to-Video Generation

2025-01-06

AI Summary

To address inaccurate motion modeling, poor temporal coherence, and weak text-video alignment in multi-object image-to-video generation, this paper proposes a two-stage diffusion-based framework. First, it generates mask-driven motion trajectory representations that explicitly encode the joint semantic-motion structure of individual objects. Second, it synthesizes high-fidelity videos conditioned on these representations. Our key contributions are: (i) the first introduction of masked motion trajectories as a compact, semantics-aware intermediate representation; and (ii) the design of object-level masked cross-attention and masked spatiotemporal self-attention mechanisms, enabling fine-grained motion control and cross-frame consistency modeling. Evaluated on multi-object high-dynamics benchmarks and our newly introduced MotionBench, the method achieves state-of-the-art performance, significantly improving temporal coherence, motion realism, and text-prompt fidelity.

Technology Category

• Computer Vision: Motion & Tracking
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

We consider the task of Image-to-Video (I2V) generation, which involves transforming static images into realistic video sequences based on a textual description. While recent advancements produce photorealistic outputs, they frequently struggle to create videos with accurate and consistent object motion, especially in multi-object scenarios. To address these limitations, we propose a two-stage compositional framework that decomposes I2V generation into: (i) An explicit intermediate representation generation stage, followed by (ii) A video generation stage that is conditioned on this representation. Our key innovation is the introduction of a mask-based motion trajectory as an intermediate representation, that captures both semantic object information and motion, enabling an expressive but compact representation of motion and semantics. To incorporate the learned representation in the second stage, we utilize object-level attention objectives. Specifically, we consider a spatial, per-object, masked-cross attention objective, integrating object-specific prompts into corresponding latent space regions and a masked spatio-temporal self-attention objective, ensuring frame-to-frame consistency for each object. We evaluate our method on challenging benchmarks with multi-object and high-motion scenarios and empirically demonstrate that the proposed method achieves state-of-the-art results in temporal coherence, motion realism, and text-prompt faithfulness. Additionally, we introduce enchmark, a new challenging benchmark for single-object and multi-object I2V generation, and demonstrate our method's superiority on this benchmark. Project page is available at https://guyyariv.github.io/TTM/.

Problem

• Image-to-Video Conversion
• Dynamic Object Rendering
• Consistency with Description

Innovation

• I2V Transformation
• Occlusion-based Motion Trajectories
• Coherent Attention Mechanism