EGVD: Event-Guided Video Diffusion Model for Physically Realistic Large-Motion Frame Interpolation

2025-03-26

AI Summary

To address performance bottlenecks in video frame interpolation (VFI) under large-motion scenarios—primarily caused by motion blur and motion ambiguity—this paper proposes a novel framework integrating high-temporal-resolution event camera signals with priors from pre-trained video diffusion models. Our key contributions are: (1) a Multimodal Motion-Conditioned Generator (MMCG), the first to enable cross-modal feature alignment and dynamic fusion between RGB frames and event streams; (2) an event-guided selective fine-tuning strategy to mitigate overfitting under limited data and complex motion; and (3) input-output normalization coupled with noise-adaptive scheduling to enhance diffusion model training stability. Built upon the Stable Video Diffusion architecture, our method achieves 27.4% and 24.1% LPIPS reduction on the Prophesee and BSRGB datasets, respectively, significantly improving physical plausibility, detail fidelity, and perceptual realism of interpolated frames—especially in large-motion and low-light conditions.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Security and Privacy: Large-scale security measurements

Abstract

Video frame interpolation (VFI) in scenarios with large motion remains challenging due to motion ambiguity between frames. While event cameras can capture high temporal resolution motion information, existing event-based VFI methods struggle with limited training data and complex motion patterns. In this paper, we introduce Event-Guided Video Diffusion Model (EGVD), a novel framework that leverages the powerful priors of pre-trained stable video diffusion models alongside the precise temporal information from event cameras. Our approach features a Multi-modal Motion Condition Generator (MMCG) that effectively integrates RGB frames and event signals to guide the diffusion process, producing physically realistic intermediate frames. We employ a selective fine-tuning strategy that preserves spatial modeling capabilities while efficiently incorporating event-guided temporal information. We incorporate input-output normalization techniques inspired by recent advances in diffusion modeling to enhance training stability across varying noise levels. To improve generalization, we construct a comprehensive dataset combining both real and simulated event data across diverse scenarios. Extensive experiments on both real and simulated datasets demonstrate that EGVD significantly outperforms existing methods in handling large motion and challenging lighting conditions, achieving substantial improvements in perceptual quality metrics (27.4% better LPIPS on Prophesee and 24.1% on BSRGB) while maintaining competitive fidelity measures. Code and datasets available at: https://github.com/OpenImagingLab/EGVD.

Problem

• Addresses video frame interpolation challenges with large motion ambiguity
• Integrates event camera data with diffusion models for realistic interpolation
• Improves generalization using combined real and simulated event datasets

Innovation

• Leverages pre-trained stable video diffusion models
• Integrates RGB and event signals via MMCG
• Uses selective fine-tuning for temporal information