LongE2V: Long-Horizon Event-based Video Reconstruction, Prediction, and Frame Interpolation with Video Diffusion Models

2026-07-09

AI Summary

This work addresses the challenges of texture blurriness and temporal inconsistency in reconstructing high-quality videos from sparse event streams. The authors propose a unified framework based on a pre-trained video diffusion model, fine-tuned to jointly perform event-based video reconstruction, prediction, and frame interpolation. Key innovations include an autoregressive unrolling strategy with adaptive context switching to mitigate temporal drift in long sequences, recoding alignment combined with cross-residual correction to ensure bidirectional consistency in interpolation, and event voxel density augmentation to enhance robustness across resolutions. Evaluated on real-world benchmarks, the method significantly outperforms existing approaches across all three tasks, demonstrating superior temporal coherence and strong zero-shot generalization capabilities.

Abstract

Recovering high-quality video from sparse event streams is a challenging task. Regression methods often blur textures, while existing generative models struggle with long-term stability. We propose LongE2V, a novel approach that leverages pre-trained video diffusion priors to jointly handle event-based video reconstruction, prediction, and frame interpolation. By fine-tuning a foundational video model, our approach achieves high data efficiency and superior perceptual quality. We introduce Autoregressive Unrolling and Adaptive Context Switching to mitigate temporal drift in extremely long sequences. We also propose Reencoding Alignment with Cross Residual Correction to ensure precise bidirectional consistency during frame interpolation. Furthermore, Event Voxel Density Augmentation ensures robustness across varying sensor resolutions. Extensive experiments on real-world benchmarks demonstrate that LongE2V outperforms state-of-the-art methods across all three tasks, exhibiting exceptional temporal coherence and zero-shot generalization. Project page: https://cdfan0627.github.io/LongE2V-page/

Problem

• event-based video
• video reconstruction
• frame interpolation
• long-horizon prediction
• temporal coherence

Innovation

• video diffusion models
• event-based vision
• temporal coherence
• frame interpolation
• long-horizon video generation