Removing Cost Volumes from Optical Flow Estimators

2025-10-15

AI Summary

Cost volumes in optical flow estimation incur substantial computational and memory overhead, severely limiting inference speed and resolution scalability. To address this, we propose a novel cost-volume-free optical flow estimation paradigm built upon the RAFT architecture. Our method introduces a progressive training strategy that dynamically sparsifies the cost volume during training and ultimately eliminates it entirely, while jointly incorporating lightweight module design and knowledge distillation. This is the first end-to-end, cost-volume-free approach achieving state-of-the-art accuracy. Crucially, it delivers significant efficiency gains: the optimal variant runs 1.2× faster and reduces GPU memory consumption by 6× compared to prior cost-volume-based methods; the fastest variant achieves real-time inference at 20 FPS on Full HD (1920×1080) resolution using only 500 MB of GPU memory.

Technology Category

• Computer Vision: Learning & Optimization for CV
• Search and Optimization: Learning to Search
• Intelligent Robots: Learning & Optimization for ROB

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Cost volumes are used in every modern optical flow estimator, but due to their computational and space complexity, they are often a limiting factor regarding both processing speed and the resolution of input frames. Motivated by our empirical observation that cost volumes lose their importance once all other network parts of, e.g., a RAFT-based pipeline have been sufficiently trained, we introduce a training strategy that allows removing the cost volume from optical flow estimators throughout training. This leads to significantly improved inference speed and reduced memory requirements. Using our training strategy, we create three different models covering different compute budgets. Our most accurate model reaches state-of-the-art accuracy while being $1.2 imes$ faster and having a $6 imes$ lower memory footprint than comparable models; our fastest model is capable of processing Full HD frames at $20,mathrm{FPS}$ using only $500,mathrm{MB}$ of GPU memory.

Problem

• Eliminating cost volumes from optical flow networks
• Reducing computational complexity and memory usage
• Improving inference speed while maintaining accuracy

Innovation

• Training strategy removes cost volumes from optical flow
• Creates models with improved speed and reduced memory
• Most accurate model achieves state-of-the-art performance efficiently