On the Real-World Generalisability of Optical Flow Models

2026-07-11

AI Summary

Current optical flow models excel on synthetic benchmarks but lack systematic evaluation of their generalization to real-world scenarios. To address this gap, this work introduces FlowFactor, the first fine-grained optical flow evaluation benchmark tailored for real-world conditions, comprising 8,204 pairs of human-annotated real video frames. The benchmark enables controlled testing across four key disturbance factors: large displacements, repetitive textures, occlusions, and illumination changes. Zero-shot evaluation of prominent pre-trained models reveals that synthetic benchmarks such as Sintel and KITTI exhibit limited predictive power for real-world performance, with illumination variations and large displacements emerging as critical challenges to generalization. Moreover, merely scaling up training data proves insufficient to bridge the domain gap. This study thus exposes fundamental limitations of existing optical flow methods in practical settings and underscores the need for new evaluation paradigms and research directions.

Abstract

Real-world deployment of vision models to broadly benefit society is arguably a main research objective. In optical flow, however, the difficulty to obtain the ground truth has focused research mainly on synthetic data and domain-specific benchmarks. Here, we investigate the severity of this mismatch. We study how well modern optical flow estimation models generalise to real-world video and question if accuracy on synthetic benchmark proxies actually predicts accuracy on real-world optical flow. To address this, we build a real-world evaluation benchmark and evaluate the real-world generalisability of a broad set of recent optical flow models using standard checkpoints. Our benchmark contains 8,204 frame pairs across TAP-Flow, Slow Flow, and our own dataset FlowFactor. FlowFactor is a manually annotated real-world benchmark of 1,000 HD frame pairs organised into four confounding factors: large displacements, repetitive textures, occlusions, and lighting variation. Each setting mainly varies only one factor, enabling diagnostic, confounder-specific analysis. Using FlowFactor, we reveal that performance on varying lighting and large displacements correlates most strongly with real-world accuracy, and that improvements on large-motion regimes can trade off against robustness in small-motion, stationary scenes. Our experiments show that progress on Sintel, KITTI and Spring only weakly predicts accuracy on real-world data, highlighting the need for a broad real-world optical flow benchmark. Interestingly, scaling up the amount of training data does not necessarily resolve the gap, calling for new innovative research instead of simply scaling data and compute.

Problem

• optical flow
• real-world generalisability
• synthetic benchmarks
• domain gap
• evaluation benchmark

Innovation

• optical flow
• real-world generalisation
• FlowFactor
• benchmarking
• domain gap