High-Resolution Frame Interpolation with Patch-based Cascaded Diffusion

2024-10-15

AAAI Conference on Artificial Intelligence

AI Summary

Existing frame interpolation methods struggle to model large motions, repetitive textures, and slender structures in high-resolution videos. To address this, we propose HiFI—the first image-patch-based cascaded pixel diffusion framework. HiFI employs a single-model, multi-scale patch-cascaded diffusion mechanism that jointly captures global motion coherence and local detail fidelity, circumventing memory explosion caused by conventional cascaded spatial upscaling. Notably, HiFI is the first method to jointly optimize frame interpolation and spatial super-resolution without sequential upsampling. Extensive evaluations demonstrate state-of-the-art performance on mainstream benchmarks including Vimeo-90K, Xiph, X-Test, and SEPE-8K. Moreover, on our newly constructed challenging dataset LaMoR—designed to stress-test motion modeling and structural preservation—HiFI significantly outperforms existing approaches, establishing new performance ceilings for high-fidelity video interpolation.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Despite the recent progress, existing frame interpolation methods still struggle with processing extremely high resolution input and handling challenging cases such as repetitive textures, thin objects, and large motion. To address these issues, we introduce a patch-based cascaded pixel diffusion model for high resolution frame interpolation, HiFI, that excels in these scenarios while achieving competitive performance on standard benchmarks. Cascades, which generate a series of images from low to high resolution, can help significantly with large or complex motion that require both global context for a coarse solution and detailed context for high resolution output. However, contrary to prior work on cascaded diffusion models which perform diffusion on increasingly large resolutions, we use a single model that always performs diffusion at the same resolution and upsamples by processing patches of the inputs and the prior solution. At inference time, this drastically reduces memory usage and allows a single model, solving both frame interpolation (base model’s task) and spatial up-sampling, saving training cost as well. HiFI excels at high-resolution images and complex repeated textures that require global context, achieving comparable or state-of-the-art performance on various benchmarks (Vimeo, Xiph, X-Test, and SEPE-8K). We further introduce a new dataset, LaMoR, that focuses on particularly challenging cases, and HiFI significantly outperforms other baselines.

Problem

• Handles high-resolution frame interpolation challenges
• Addresses repetitive textures and large motion issues
• Reduces memory usage with patch-based diffusion

Innovation

• Patch-based cascaded diffusion for high resolution
• Single model handles diffusion and upsampling
• Reduces memory usage and training cost