Event-based Video Frame Interpolation with Cross-Modal Asymmetric Bidirectional Motion Fields

2023-06-01

Computer Vision and Pattern Recognition

AI Summary

Existing video frame interpolation (VFI) methods struggle to model realistic asymmetric, high-speed, and texture-rich motion in event-camera scenarios. To address this, we propose the first cross-modal asymmetric bidirectional motion field estimation framework for event-driven VFI: leveraging the complementary nature of event streams and intensity frames, it directly regresses asymmetric optical flow. We design EIF-BiOFNet—a novel neural architecture—and an interactive attention-based fusion module; additionally, we introduce ERF-X, the first high-frame-rate (170 FPS) event-based VFI benchmark featuring extreme motion and dynamic textures. Extensive experiments demonstrate that our method significantly outperforms state-of-the-art approaches across multiple datasets, achieving substantial PSNR and SSIM improvements—particularly under high-speed motion and complex texture conditions.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Video Frame Interpolation (VFI) aims to generate intermediate video frames between consecutive input frames. Since the event cameras are bio-inspired sensors that only encode brightness changes with a micro-second temporal resolution, several works utilized the event camera to enhance the performance of VFI. However, existing methods estimate bidirectional inter-frame motion fields with only events or approximations, which can not consider the complex motion in real-world scenarios. In this paper, we propose a novel event-based VFI framework with crossmodal asymmetric bidirectional motion field estimation. In detail, our EIF-BiOFNet utilizes each valuable characteristic of the events and images for direct estimation of inter-frame motion fields without any approximation methods. Moreover, we develop an interactive attention-based frame synthesis network to efficiently leverage the complementary warping-based and synthesis-based features. Finally, we build a large-scale event-based VFI dataset, ERF-X170FPS, with a high frame rate, extreme motion, and dynamic textures to overcome the limitations of previous event-based VFI datasets. Extensive experimental results validate that our method shows significant performance improvement over the state-of-the-art VFI methods on various datasets. Our project pages are available at: https://github.com/intelpro/CBMNet

Problem

• Enhances video frame interpolation accuracy
• Utilizes event cameras for motion estimation
• Creates high-frame-rate VFI dataset

Innovation

• Cross-modal asymmetric bidirectional motion fields
• Interactive attention-based frame synthesis network
• Large-scale event-based VFI dataset