FlowMo: Variance-Based Flow Guidance for Coherent Motion in Video Generation

2025-06-01

AI Summary

Text-to-video diffusion models commonly suffer from temporal modeling deficiencies, including motion discontinuity and physically implausible dynamics. To address this, we propose a plug-and-play, fine-tuning-free motion guidance method that requires no additional conditional inputs. Our approach introduces, for the first time, an appearance-agnostic inter-frame distance metric in the latent space as a temporal representation, and designs a dynamic backward guidance mechanism based on block-level temporal variance to enhance motion consistency during sampling. The method operates solely via lightweight analysis of pre-trained model latent variables—requiring no architectural modifications or retraining. Extensive evaluation across multiple state-of-the-art text-to-video models demonstrates significant improvements in motion coherence and physical plausibility, while preserving high visual fidelity and precise text-video alignment.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Text-to-video diffusion models are notoriously limited in their ability to model temporal aspects such as motion, physics, and dynamic interactions. Existing approaches address this limitation by retraining the model or introducing external conditioning signals to enforce temporal consistency. In this work, we explore whether a meaningful temporal representation can be extracted directly from the predictions of a pre-trained model without any additional training or auxiliary inputs. We introduce extbf{FlowMo}, a novel training-free guidance method that enhances motion coherence using only the model's own predictions in each diffusion step. FlowMo first derives an appearance-debiased temporal representation by measuring the distance between latents corresponding to consecutive frames. This highlights the implicit temporal structure predicted by the model. It then estimates motion coherence by measuring the patch-wise variance across the temporal dimension and guides the model to reduce this variance dynamically during sampling. Extensive experiments across multiple text-to-video models demonstrate that FlowMo significantly improves motion coherence without sacrificing visual quality or prompt alignment, offering an effective plug-and-play solution for enhancing the temporal fidelity of pre-trained video diffusion models.

Problem

• Enhancing motion coherence in video generation models
• Extracting temporal representation without retraining
• Reducing patch-wise variance for dynamic motion guidance

Innovation

• Training-free guidance for motion coherence
• Appearance-debiased temporal representation extraction
• Dynamic variance reduction during sampling