Scrutinizing Index-Based Risk Assessments: A Case Study in NYC Decision-making for Heat Emergency Management

2026-05-17

AI Summary

This study addresses the limitations of manually constructed risk indices used by governments for geographically targeted heatwave emergency response, which are highly sensitive to variable selection and spatial scale, potentially undermining decision reliability and equity. Focusing on extreme heat emergencies in New York City, the research introduces, for the first time in public-sector risk assessment, a validity and reliability framework drawn from measurement theory. Through systematic comparison—encompassing index construction, sensitivity analyses, multi-scale evaluations, and simulation experiments—the study contrasts the performance of traditional risk indices against predictive algorithms. Findings reveal that seemingly reasonable design choices can substantially alter risk scores and consequently shape policy responses. Building on these insights, the work proposes a decision-support guideline that balances transparency, robustness, and goal alignment, offering a generalizable methodological foundation for public emergency management.

Abstract

Cities are increasingly turning to large-scale data analysis and machine learning to make consequential decisions. While the algorithmic fairness community has focused on analyzing the risks and benefits associated with these complex methods, there has been much less scrutiny of the many simpler, but still widely used, data-driven tools that support government decision-making in a variety of settings. In this work, we study hand-crafted indices for geographic targeting and decision-making in emergency management -- a field responsible for coordinating preparedness and response efforts to hazards ranging from natural disasters to human threats. Indices, which capture abstract principles and overarching priorities (e.g., reducing social vulnerability), are low-complexity models that statistically aggregate chosen variables. They are generally flexible and interpretable, but can also be sensitive to key design choices and require strong assumptions. Through a case study of decision-making for extreme heat emergencies in NYC, we examine the challenges that practitioners may face in selecting an index for preparedness and response actions. We map empirical findings from index-based simulations to concerns related to validity and reliability from the measurement literature and show via sensitivity analyses that different reasonable choices of input variables or spatial scale can result in substantive differences to index risk scores, thereby affecting downstream government decision-making. We contrast these challenges with considerations for developing predictive algorithms that more narrowly relate to concrete, measurable outcomes. Ultimately, we provide generalizable recommendations that practitioners and public-sector technologists can use for navigating the trade-offs between indices and predictive algorithms in other government settings.

Problem

• index-based risk assessment
• emergency management
• algorithmic fairness
• heat emergency
• government decision-making

Innovation

• index-based risk assessment
• algorithmic fairness
• emergency management
• sensitivity analysis
• predictive algorithms