UniRED: Unified RGB-D Video Frame Interpolation with Event Guidance

2026-06-23

AI Summary

Low frame-rate RGB-D videos struggle to accurately capture fast-moving dynamic scenes, and existing frame interpolation methods often suffer from blurry boundaries, visual artifacts, and geometric inconsistencies. To address this, this work proposes the first unified multimodal frame interpolation framework that jointly leverages RGB, depth, and event camera data, uniquely exploiting the high temporal resolution of event streams alongside RGB-D information. The method achieves high-quality intermediate frame synthesis through bidirectional optical flow estimation, motion basis optimization, Z-axis depth refinement, and multimodal feature fusion. Experiments demonstrate that the proposed approach significantly outperforms state-of-the-art methods in both RGB photometric fidelity and depth geometric accuracy on public benchmarks as well as on a newly introduced RGB-D-Event trimodal dataset—the first of its kind.

Abstract

High frame-rate RGB-D videos are crucial for a variety of downstream tasks, including motion analysis, dynamic scene understanding, and 3D reconstruction. However, due to hardware and sensing constraints, practical RGB-D cameras are typically limited to low frame rates, making it difficult to capture rapid scene dynamics. Existing video interpolation methods have achieved strong performance on RGB data, but they are not readily applicable to RGB-D scenarios, where they often yield blurry boundaries, visible artifacts, and degraded geometric consistency. Furthermore, motion estimation from only two boundary frames is inherently under-constrained in complex dynamic scenes. Event cameras, by contrast, provide asynchronous measurements with ultra-high temporal resolution, offering dense motion cues. In this paper, we propose a unified multimodal framework for RGB-D video interpolation that jointly exploits RGB appearance, depth geometry, and event-based temporal cues. Specifically, it first extracts and fuses RGB, depth and event cues, then estimates bidirectional flow with motion basis refinement for RGB and Z-axial refinement for depth, and finally synthesizes the target RGB-D frame via bidirectional warping and soft blending. In addition, we construct a new RGB-D-Event dataset to alleviate the scarcity of tri-modal training data. Extensive experiments on a public benchmark and the proposed dataset demonstrate that our method achieves superior photometric fidelity for RGB interpolation and stronger geometric accuracy for depth interpolation than existing approaches.

Problem

• RGB-D video interpolation
• frame rate limitation
• geometric consistency
• motion estimation
• event cameras

Innovation

• RGB-D video interpolation
• event camera
• multimodal fusion
• geometric consistency
• motion estimation