RobustSpring: Benchmarking Robustness to Image Corruptions for Optical Flow, Scene Flow and Stereo

2025-05-14

AI Summary

Existing optical flow, scene flow, and stereo vision benchmarks evaluate only accuracy, lacking systematic assessment of robustness to realistic image degradations (e.g., noise, rain, fog, blur). Method: We introduce the first unified robustness benchmark for these three tasks, covering 20 real-world degradation types. We propose a novel time-disparity-depth-consistent degradation generation paradigm and a new robustness metric, elevating robustness to a core evaluation dimension on par with accuracy. Leveraging the high-resolution Spring dataset, we release 20,000 high-fidelity degraded images. Contribution/Results: We establish a dual-axis evaluation framework jointly measuring accuracy and robustness. Extensive experiments reveal weak correlation between accuracy and robustness, and substantial performance variation across degradation types—highlighting critical limitations of current models. This benchmark provides foundational infrastructure and actionable insights for developing robust vision models.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Standard benchmarks for optical flow, scene flow, and stereo vision algorithms generally focus on model accuracy rather than robustness to image corruptions like noise or rain. Hence, the resilience of models to such real-world perturbations is largely unquantified. To address this, we present RobustSpring, a comprehensive dataset and benchmark for evaluating robustness to image corruptions for optical flow, scene flow, and stereo models. RobustSpring applies 20 different image corruptions, including noise, blur, color changes, quality degradations, and weather distortions, in a time-, stereo-, and depth-consistent manner to the high-resolution Spring dataset, creating a suite of 20,000 corrupted images that reflect challenging conditions. RobustSpring enables comparisons of model robustness via a new corruption robustness metric. Integration with the Spring benchmark enables public two-axis evaluations of both accuracy and robustness. We benchmark a curated selection of initial models, observing that accurate models are not necessarily robust and that robustness varies widely by corruption type. RobustSpring is a new computer vision benchmark that treats robustness as a first-class citizen to foster models that combine accuracy with resilience. It will be available at https://spring-benchmark.org.

Problem

• Assessing model robustness to image corruptions in optical flow, scene flow, and stereo vision
• Lack of standardized benchmarks for real-world perturbation resilience in vision tasks
• Evaluating accuracy and robustness trade-offs across diverse corruption types

Innovation

• Comprehensive dataset for corruption robustness evaluation
• 20 diverse image corruptions applied consistently
• New metric for comparing model robustness