Fixing the Pitfalls of Probabilistic Time-Series Forecasting Evaluation by Kernel Quadrature

2025-03-08

AI Summary

In probabilistic time-series forecasting, common estimators of the Continuous Ranked Probability Score (CRPS)—such as quantile approximation and probability-weighted moments—suffer from systematic bias, leading to inaccurate model evaluation and inconsistent ranking. To address this, we propose the first unbiased CRPS estimator based on kernel quadrature and cubature rules, marking the inaugural application of kernel quadrature to probabilistic forecast evaluation. Our method is theoretically unbiased and computationally scalable, circumventing inherent errors from discretization and moment-based approximations. Empirically, it achieves significantly higher CRPS estimation accuracy across multiple benchmark datasets and improves ranking consistency among forecasting models. It outperforms existing implementations in mainstream libraries—including GluonTS—across all evaluated metrics, establishing a new state-of-the-art for CRPS estimation in probabilistic forecasting.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Probabilistic Programming
• Intelligent Robots: State Estimation

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Despite the significance of probabilistic time-series forecasting models, their evaluation metrics often involve intractable integrations. The most widely used metric, the continuous ranked probability score (CRPS), is a strictly proper scoring function; however, its computation requires approximation. We found that popular CRPS estimators--specifically, the quantile-based estimator implemented in the widely used GluonTS library and the probability-weighted moment approximation--both exhibit inherent estimation biases. These biases lead to crude approximations, resulting in improper rankings of forecasting model performance when CRPS values are close. To address this issue, we introduced a kernel quadrature approach that leverages an unbiased CRPS estimator and employs cubature construction for scalable computation. Empirically, our approach consistently outperforms the two widely used CRPS estimators.

Problem

• Addresses biases in CRPS estimators for time-series forecasting.
• Improves accuracy of probabilistic forecasting model evaluations.
• Introduces kernel quadrature for unbiased CRPS estimation.

Innovation

• Kernel quadrature for unbiased CRPS estimation
• Cubature construction for scalable computation
• Outperforms popular CRPS estimators empirically