Benchmarking the Alignment of Data-Quality Metrics, Human Judgment and Land-Cover Segmentation Performance for Earth Observation

2026-06-23

AI Summary

This study addresses the limitations of conventional image quality metrics—such as FID, SSIM, KID, IS, and LPIPS—in evaluating synthetic remote sensing imagery, demonstrating their poor alignment with both human perception and downstream task performance. By generating Earth observation images using deep generative models, the work systematically compares automatic metrics against human judgments and semantic segmentation accuracy for land cover classification. The findings reveal a significant misalignment: semantic-preserving perturbations substantially alter automatic scores without affecting human interpretability, and low-scoring synthetic samples can enhance segmentation performance when used in mixed training regimes. Moreover, metrics based on ImageNet feature spaces prove unreliable for geospatial data, underscoring the necessity of grounding synthetic data evaluation in task-specific performance and human assessment rather than generic image fidelity measures.

Abstract

Volume and quality of datasets are crucial for deep learning model training, yet they are often constrained by availability and data acquisition costs. Synthetic data augmentation can extend existing datasets with realistic images, and the quality of these images is generally assessed through fidelity metrics such as FID, KID, IS, LPIPS and SSIM that measure structural or distributional similarity. However, such metrics, including the widely used FID, focus on visual fidelity without reflecting downstream utility, and can diverge from human perception under perturbations that are imperceptible to human observers. In this work, we systematically evaluate Earth observation datasets alongside synthetic counterparts generated by deep generative models, comparing automatic metrics against human perception and downstream tasks. Our results reveal a stark misalignment: semantics-preserving perturbations such as rotation drastically alter metric scores while leaving human recognition unaffected, and synthetic samples that score poorly on automatic metrics achieve comparable or higher perceived realism, and can improve downstream performance when combined with real data. By benchmarking semantic segmentation models trained on mixed real-synthetic datasets, we demonstrate that quality metrics rooted in ImageNet-pretrained feature spaces are unreliable indicators for geospatial data. Our findings underscore that automatic quality evaluation of synthetic datasets should be grounded in downstream task performance and human evaluation.

Problem

• synthetic data
• quality metrics
• human perception
• downstream task performance
• Earth observation

Innovation

• synthetic data
• quality metrics
• human perception
• downstream task performance
• Earth observation