🤖 AI Summary
This study addresses the lack of consistent evaluation metrics and poor comparability in existing machine learning research for monitoring the United Nations Sustainable Development Goals (SDGs) using remote sensing data. Applying the PRISMA protocol, the authors systematically analyzed 86 experiments from 20 studies through a three-level random-effects meta-regression with double arcsine transformation. The overall mean accuracy was found to be 0.90 [0.86, 0.92], yet model performance exhibited high sensitivity to class imbalance. Notably, 64% of the variance in performance stemmed from between-study differences, with the proportion of the majority class alone accounting for 61% of this heterogeneity. To enhance cross-study comparability and advance standardized evaluation practices in SDG monitoring, the authors advocate for the mandatory reporting of confusion matrices in future studies.
📝 Abstract
Machine learning (ML) is a tool to exploit remote sensing data for the monitoring and implementation of the United Nations'Sustainable Development Goals (SDGs). In this paper, we report on a meta-analysis to evaluate the performance of ML applied to remote sensing data to monitor SDGs. Specifically, we aim to 1) estimate the average performance; 2) determine the degree of heterogeneity between and within studies; and 3) assess how study features influence model performance. Using PRISMA guidelines, a search was performed across multiple academic databases to identify potentially relevant studies. A random sample of 200 was screened by three reviewers, resulting in 86 trials within 20 studies with 14 study features. Overall accuracy was the most reported performance metric. It was analyzed using double arcsine transformation and a three-level random effects model. The average overall accuracy of the best model was 0.90 [0.86, 0.92]. There was considerable heterogeneity in model performance, 64% of which was between studies. The only significant feature was the prevalence of the majority class, which explained 61% of the between-study heterogeneity. None of the other thirteen features added value to the model. The most important contributions of this paper are the following two insights. 1) Overall accuracy is the most popular performance metric, yet arguably the least insightful. Its sensitivity to class imbalance makes it necessary to normalize it, which is far from common practice. 2) The field needs to standardize the reporting. Reporting of the confusion matrix for independent test sets is the most important ingredient for between-study comparisons of ML classifiers. These findings underscore the need for robust and comparable evaluation metrics in machine learning applications to ensure reliable and actionable insights for effective SDG monitoring and policy formulation.