A Framework for Evaluating PM2.5 Forecasts from the Perspective of Individual Decision Making

2024-09-09

arXiv.org

AI Summary

This study evaluates the effectiveness of hourly PM₂.₅ forecasts across the contiguous United States in supporting individual-level wildfire smoke exposure avoidance decisions—specifically, “whether” and “when” to go outdoors. Addressing the limitations of conventional forecast evaluation paradigms centered on statistical error metrics, we propose a decision-aware loss function and establish the first evaluation framework explicitly designed for personal protective decision-making. Integrating multi-source meteorological and air quality data, we employ machine learning models and develop an interpretable visualization toolkit alongside an open-source benchmarking codebase. Results reveal substantial deficiencies in leading operational forecasts under realistic decision-making scenarios. Our work delivers a standardized, decision-centric evaluation protocol and publicly accessible resources, advancing the paradigm shift in air quality forecasting—from statistical accuracy toward decision utility.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Sequential Decision Making
• Humans and AI: Planning and Decision Support for Human-Machine Teams

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Wildfire frequency is increasing as the climate changes, and the resulting air pollution poses health risks. Just as people routinely use weather forecasts to plan their activities around precipitation, reliable air quality forecasts could help individuals reduce their exposure to air pollution. In the present work, we evaluate several existing forecasts of fine particular matter (PM2.5) within the continental United States in the context of individual decision-making. Our comparison suggests there is meaningful room for improvement in air pollution forecasting, which might be realized by incorporating more data sources and using machine learning tools. To facilitate future machine learning development and benchmarking, we set up a framework to evaluate and compare air pollution forecasts for individual decision making. We introduce a new loss to capture decisions about when to use mitigation measures. We highlight the importance of visualizations when comparing forecasts. Finally, we provide code to download and compare archived forecast predictions.

Problem

• Evaluating accuracy of hourly PM2.5 forecasts for wildfire smoke exposure
• Assessing forecast performance for personal outdoor activity planning decisions
• Identifying improvement needs in PM2.5 prediction models and methods

Innovation

• Evaluated six existing PM2.5 forecasting methods
• Introduced new evaluation metric for exposure timing
• Combined physical simulation, ensembling, and AI techniques