IDOBE: Infectious Disease Outbreak forecasting Benchmark Ecosystem

2026-04-20

AI Summary

This study addresses the lack of standardized benchmarks and limited understanding of method performance in forecasting emerging infectious disease outbreaks. The authors introduce the first standardized time-series benchmark dataset spanning over a century, multiple regions, and 13 diseases, generating more than 10,000 outbreak events via derivative-based segmentation to enable multi-step short-term prediction evaluation. Epidemiological diversity is characterized using information-theoretic and distributional metrics. A systematic assessment is conducted using 11 baseline models—including MLPs and statistical approaches—evaluated through both point forecasts and probabilistic scoring rules such as the normalized weighted interval score (NWIS). Results indicate that MLP-based methods are generally the most robust, while statistical models show slight advantages in predicting peaks. The dataset and code are publicly released to foster reproducible research.

Technology Category

• Machine Learning: Time-Series/Data Streams
• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Epidemic forecasting has become an integral part of real-time infectious disease outbreak response. While collaborative ensembles composed of statistical and machine learning models have become the norm for real-time forecasting, standardized benchmark datasets for evaluating such methods are lacking. Further, there is limited understanding on performance of these methods for novel outbreaks with limited historical data. In this paper, we propose IDOBE, a curated collection of epidemiological time series focused on outbreak forecasting. IDOBE compiles from multiple data repositories spanning over a century of surveillance and across U.S. states and global locations. We perform derivative-based segmentation to generate over 10,000 outbreaks covering multiple outcomes such as cases and hospitalizations for 13 diseases. We consider a variety of information-theoretic and distributional measures to quantify the epidemiological diversity of the dataset. Finally, we perform multi-horizon short-term forecasting (1- to 4-week-ahead) through the progression of the outbreak using 11 baseline models and report on their performance. In addition to standard metrics such as NMSE and MAPE for point forecasts, we include probabilistic scoring rules such as Normalized Weighted Interval Score (NWIS) to quantify the performance. We find that MLP-based methods have the most robust performance, with statistical methods having a slight edge during the pre-peak phase. IDOBE dataset along with baselines are released publicly on https://github.com/NSSAC/IDOBE to enable standardized, reproducible benchmarking of outbreak forecasting methods.

Problem

• epidemic forecasting
• benchmark dataset
• outbreak prediction
• limited historical data
• standardized evaluation

Innovation

• outbreak forecasting
• benchmark dataset
• epidemiological time series
• derivative-based segmentation
• probabilistic scoring