MotiF: Making Text Count in Image Animation with Motion Focal Loss

2024-12-20

arXiv.org

AI Summary

Existing text-to-image-to-video (TI2V) methods struggle to accurately model motion semantics described in text prompts, resulting in poor alignment between generated videos and motion specifications. To address this, we propose a Motion-Focused Loss that leverages optical flow estimation to construct motion heatmaps, enabling spatially weighted optimization of the noise prediction process in diffusion models—thereby significantly improving dynamic semantic consistency. Furthermore, we introduce TI2V Bench, the first dedicated benchmark for TI2V evaluation, comprising 320 text–image pairs and an accompanying human preference assessment protocol. On this benchmark, our method outperforms nine leading open-source models, achieving a human preference rate of 72%. All code, datasets, and comprehensive results are publicly released.

Technology Category

• Computer Vision: Motion & Tracking
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Text-Image-to-Video (TI2V) generation aims to generate a video from an image following a text description, which is also referred to as text-guided image animation. Most existing methods struggle to generate videos that align well with the text prompts, particularly when motion is specified. To overcome this limitation, we introduce MotiF, a simple yet effective approach that directs the model's learning to the regions with more motion, thereby improving the text alignment and motion generation. We use optical flow to generate a motion heatmap and weight the loss according to the intensity of the motion. This modified objective leads to noticeable improvements and complements existing methods that utilize motion priors as model inputs. Additionally, due to the lack of a diverse benchmark for evaluating TI2V generation, we propose TI2V Bench, a dataset consists of 320 image-text pairs for robust evaluation. We present a human evaluation protocol that asks the annotators to select an overall preference between two videos followed by their justifications. Through a comprehensive evaluation on TI2V Bench, MotiF outperforms nine open-sourced models, achieving an average preference of 72%. The TI2V Bench and additional results are released in https://wang-sj16.github.io/motif/.

Problem

• Improves text-video alignment in image animation
• Enhances motion generation via motion focal loss
• Introduces benchmark for robust TI2V evaluation

Innovation

• Uses optical flow for motion heatmap
• Applies motion-weighted focal loss
• Introduces TI2V Bench dataset