MEMFOF: High-Resolution Training for Memory-Efficient Multi-Frame Optical Flow Estimation

2025-06-29

AI Summary

Optical flow estimation methods often suffer from GPU memory explosion at high resolutions (e.g., 1080p) due to prohibitively large correlation volumes, hindering end-to-end training and deployment. To address this, we propose MEMFOF—the first efficient framework enabling native end-to-end multi-frame optical flow training and inference at full 1080p resolution without cropping or downsampling. Its core contributions are: (1) a streamlined correlation volume design that drastically reduces memory footprint; (2) a high-resolution-aware temporal multi-frame fusion mechanism; and (3) RAFT architecture enhancements coupled with system-level memory optimization strategies. MEMFOF achieves state-of-the-art performance on Spring, Sintel (clean), and KITTI-2015 benchmarks, attaining 3.289% 1-pixel error rate, 0.963 average end-point error (EPE), and 2.94% Fl-all error—while consuming significantly less GPU memory than existing methods, thus jointly optimizing accuracy and efficiency.

Technology Category

• Computer Vision: Learning & Optimization for CV
• Machine Learning: Hardware-aware ML
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Recent advances in optical flow estimation have prioritized accuracy at the cost of growing GPU memory consumption, particularly for high-resolution (FullHD) inputs. We introduce MEMFOF, a memory-efficient multi-frame optical flow method that identifies a favorable trade-off between multi-frame estimation and GPU memory usage. Notably, MEMFOF requires only 2.09 GB of GPU memory at runtime for 1080p inputs, and 28.5 GB during training, which uniquely positions our method to be trained at native 1080p without the need for cropping or downsampling. We systematically revisit design choices from RAFT-like architectures, integrating reduced correlation volumes and high-resolution training protocols alongside multi-frame estimation, to achieve state-of-the-art performance across multiple benchmarks while substantially reducing memory overhead. Our method outperforms more resource-intensive alternatives in both accuracy and runtime efficiency, validating its robustness for flow estimation at high resolutions. At the time of submission, our method ranks first on the Spring benchmark with a 1-pixel (1px) outlier rate of 3.289, leads Sintel (clean) with an endpoint error (EPE) of 0.963, and achieves the best Fl-all error on KITTI-2015 at 2.94%. The code is available at https://github.com/msu-video-group/memfof.

Problem

• Reduces GPU memory usage for high-resolution optical flow
• Balances multi-frame estimation with memory efficiency
• Achieves state-of-the-art accuracy without cropping or downsampling

Innovation

• Memory-efficient multi-frame optical flow method
• Reduced correlation volumes for lower GPU usage
• High-resolution training without cropping or downsampling