Frame Guidance: Training-Free Guidance for Frame-Level Control in Video Diffusion Models

2025-06-08

AI Summary

This work addresses the challenge of fine-grained, frame-level controllable generation in video diffusion models. We propose a training-free, inference-only frame-level guidance framework that requires neither model training nor fine-tuning. Our method introduces a novel training-free latent-space mechanism for per-frame signal injection, coupled with lightweight latent reweighting and cross-frame consistency optimization—enabling precise control over keyframes, sketches, style reference images, and depth maps, all without modifying model parameters. Crucially, we achieve this via gradient-free latent iteration, balancing control fidelity with global temporal coherence. Experiments demonstrate high-quality, controllable video generation across keyframe-guided synthesis, style transfer, and video looping tasks. The framework is fully compatible with arbitrary pre-trained video diffusion models, incurs zero training overhead, and significantly lowers deployment barriers.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Advancements in diffusion models have significantly improved video quality, directing attention to fine-grained controllability. However, many existing methods depend on fine-tuning large-scale video models for specific tasks, which becomes increasingly impractical as model sizes continue to grow. In this work, we present Frame Guidance, a training-free guidance for controllable video generation based on frame-level signals, such as keyframes, style reference images, sketches, or depth maps. For practical training-free guidance, we propose a simple latent processing method that dramatically reduces memory usage, and apply a novel latent optimization strategy designed for globally coherent video generation. Frame Guidance enables effective control across diverse tasks, including keyframe guidance, stylization, and looping, without any training, compatible with any video models. Experimental results show that Frame Guidance can produce high-quality controlled videos for a wide range of tasks and input signals.

Problem

• Enabling fine-grained control in video diffusion models without training
• Reducing memory usage for frame-level guidance in video generation
• Achieving globally coherent video generation with diverse input signals

Innovation

• Training-free guidance for frame-level control
• Latent processing reduces memory usage
• Latent optimization for coherent video generation