FreqPrior: Improving Video Diffusion Models with Frequency Filtering Gaussian Noise

2025-02-05

AI Summary

Current text-to-video generation methods suffer from motion distortion and detail loss due to inadequate modeling of noise priors in the frequency domain. To address this, we propose a frequency-aware noise initialization and optimization framework. First, we establish a theoretical model characterizing the decay of noise variance across frequency bands. Second, we design a frequency-selective filter that preserves standard Gaussian properties while enabling structured spectral control. Third, we introduce a partial perturbation sampling strategy for latent variables at intermediate timesteps, accelerating inference without compromising output quality. Evaluated on the VBench benchmark, our method achieves state-of-the-art performance in both image fidelity and semantic alignment, ranking first in overall score. Moreover, it significantly improves inference speed—up to 2.1× faster than prior approaches—while maintaining superior visual quality and temporal coherence.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Text-driven video generation has advanced significantly due to developments in diffusion models. Beyond the training and sampling phases, recent studies have investigated noise priors of diffusion models, as improved noise priors yield better generation results. One recent approach employs the Fourier transform to manipulate noise, marking the initial exploration of frequency operations in this context. However, it often generates videos that lack motion dynamics and imaging details. In this work, we provide a comprehensive theoretical analysis of the variance decay issue present in existing methods, contributing to the loss of details and motion dynamics. Recognizing the critical impact of noise distribution on generation quality, we introduce FreqPrior, a novel noise initialization strategy that refines noise in the frequency domain. Our method features a novel filtering technique designed to address different frequency signals while maintaining the noise prior distribution that closely approximates a standard Gaussian distribution. Additionally, we propose a partial sampling process by perturbing the latent at an intermediate timestep during finding the noise prior, significantly reducing inference time without compromising quality. Extensive experiments on VBench demonstrate that our method achieves the highest scores in both quality and semantic assessments, resulting in the best overall total score. These results highlight the superiority of our proposed noise prior.

Problem

• Enhance video diffusion models
• Address frequency domain noise
• Improve motion dynamics and details

Innovation

• Frequency domain noise refinement
• Gaussian distribution approximation
• Intermediate timestep partial sampling