Benchmarking Object Detectors under Real-World Distribution Shifts in Satellite Imagery

2025-03-24

AI Summary

Existing object detectors lack standardized evaluation of real-world domain generalization (DG) on satellite imagery: performance degrades significantly under source–target distribution shifts—such as across climate zones or disaster-affected vs. non-affected geographic regions—and no benchmark exists for such realistic DG assessment. To address this, we propose RWDS, the first real-world DG benchmark for satellite image object detection. RWDS systematically models two types of spatial distribution shifts—cross-climate-zone and cross-disaster-region—enabling robustness evaluation in humanitarian and climate-related applications. We construct three novel DG datasets from multi-source, high-resolution satellite imagery, accompanied by a standardized detection protocol—including mean Average Precision (mAP) and out-of-distribution (OOD) generalization error analysis—as well as open-source code. RWDS fills a critical gap in standardized, realistic DG evaluation for detection models and establishes a reproducible, comparable foundation for studying generalization under authentic distribution shifts.

Technology Category

• Computer Vision: Remote Sensing / Geospatial AI
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Application Domains: Internet of Things, Sensor Networks & Smart Cities

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Object detectors have achieved remarkable performance in many applications; however, these deep learning models are typically designed under the i.i.d. assumption, meaning they are trained and evaluated on data sampled from the same (source) distribution. In real-world deployment, however, target distributions often differ from source data, leading to substantial performance degradation. Domain Generalisation (DG) seeks to bridge this gap by enabling models to generalise to Out-Of-Distribution (OOD) data without access to target distributions during training, enhancing robustness to unseen conditions. In this work, we examine the generalisability and robustness of state-of-the-art object detectors under real-world distribution shifts, focusing particularly on spatial domain shifts. Despite the need, a standardised benchmark dataset specifically designed for assessing object detection under realistic DG scenarios is currently lacking. To address this, we introduce Real-World Distribution Shifts (RWDS), a suite of three novel DG benchmarking datasets that focus on humanitarian and climate change applications. These datasets enable the investigation of domain shifts across (i) climate zones and (ii) various disasters and geographic regions. To our knowledge, these are the first DG benchmarking datasets tailored for object detection in real-world, high-impact contexts. We aim for these datasets to serve as valuable resources for evaluating the robustness and generalisation of future object detection models. Our datasets and code are available at https://github.com/RWGAI/RWDS.

Problem

• Assessing object detectors under real-world distribution shifts
• Lacking standardized benchmarks for domain generalization scenarios
• Introducing datasets for climate zones and disaster regions

Innovation

• Domain Generalisation for OOD robustness
• Real-World Distribution Shifts datasets
• Benchmarking spatial domain shifts