Moving beyond spatial and random cross-validation in environmental modelling: a call for prediction-domain adaptive evaluation

2026-05-13

AI Summary

This study addresses the limitations of conventional cross-validation methods in accurately capturing the complex spatial relationships between training data and target prediction regions, which often leads to biased model performance estimates. To bridge the methodological gap between random and spatial cross-validation, the authors propose a novel paradigm termed “prediction-domain adaptive evaluation.” This framework dynamically tailors the cross-validation strategy to align with the actual prediction scenario by integrating spatial statistics with machine learning evaluation techniques, thereby enabling an adaptive validation workflow. Extensive simulations demonstrate the robustness of the approach across a continuum from interpolation to extrapolation settings. Empirical results show that the proposed method consistently yields more reliable and accurate estimates of predictive accuracy under diverse data distributions.

Abstract

With the growing application of spatial predictive modeling in ecology, the question of how to appropriately evaluate the resulting maps has gained increasing attention. While there is consensus that map accuracy is ideally estimated using an independent probability sample of the prediction area, there is still no agreement on the most appropriate way to conduct an evaluation for the common case when such a sample is not available. Cross-validation, which involves multiple train-test splits, is commonly applied not only to estimate final model accuracy but also to guide model tuning and selection. Many different spatial and non-spatial approaches to cross-validation have been proposed, and approaches in both groups have faced substantial criticism. It has been shown that random cross-validation methods are suitable when the training points are randomly distributed in the prediction area, while spatial cross-validation is better suited towards extrapolation situations. In practice, however, there is a continuum and most cases are between those two extremes. To address this gap, we advocate for a new category of cross-validation methods to account for this: prediction-domain adaptive evaluation. Methods in this category flexibly adapt to the prediction situation, yielding most reliable estimates of map accuracy across different scenarios. To ground this perspective empirically, we reproduce a simulation study that was used in earlier research and systematically compare different evaluation methods and discuss their purpose.

Problem

• spatial predictive modeling
• cross-validation
• map accuracy
• prediction-domain evaluation
• environmental modelling

Innovation

• prediction-domain adaptive evaluation
• spatial cross-validation
• model evaluation
• environmental modelling
• accuracy estimation