Uncertainty Quantification in Forecast Comparisons

2026-05-05

AI Summary

Existing predictive evaluation methods struggle to rigorously quantify sampling uncertainty in multidimensional settings, often leading to inflated Type I error rates under multiple comparisons and invalid joint inference. This work proposes a unified statistical framework that constructs simultaneous confidence bands—applicable across multivariate, multi-step-ahead, multi-location, and multi-model configurations—for joint inference on mean, quantile, and distributional forecasts. The approach builds upon a multivariate extension of the Diebold–Mariano test and incorporates bootstrap-based uncertainty quantification. Empirical validation in macroeconomic and weather forecasting applications demonstrates the framework’s ability to effectively discern predictive advantages of time-varying parameter models against data-driven alternatives.

Abstract

Skill scores, which measure the relative improvement of a forecasting method over a benchmark via consistent scoring functions and proper scoring rules, are a standard tool in forecast evaluation, yet their sampling uncertainty is rarely rigorously quantified. With modern forecasting applications being increasingly multivariate and involving evaluations across multiple horizons, variables, spatial locations, and forecasting methods, standard tools like the pairwise Diebold-Mariano forecast accuracy test or pointwise confidence intervals fail to account for the multiple comparison problem, leading to inflated Type I error rates and invalid joint inference. To address the lack of a coherent, statistically rigorous framework for quantifying uncertainty across these multi-dimensional evaluation problems, we introduce simultaneous confidence bands for expected scores and skill scores. Our framework provides a versatile tool for joint inference that is applicable to any forecast type from mean and quantile to full distributional forecasts. We develop a bootstrap implementation and show that our bands are valid under multivariate extensions of the classical Diebold-Mariano assumptions. We demonstrate the practical utility of the approach in two case studies by quantifying the benefits of time-varying parameter models for macroeconomic forecasting, and by comparing data-driven and physics-based models in probabilistic weather forecasting.

Problem

• Uncertainty Quantification
• Forecast Comparison
• Skill Scores
• Multiple Comparisons
• Joint Inference

Innovation

• simultaneous inference
• uncertainty quantification
• skill scores
• forecast evaluation
• bootstrap