All Models Are Wrong, But Can They Be Useful? Lessons from COVID-19 Agent-Based Models: A Systematic Review

2025-09-12

AI Summary

Despite widespread adoption of agent-based models (ABMs) for public health policy during the COVID-19 pandemic, their real-world decision-support utility remains poorly characterized and empirically underassessed. Method: We conducted a systematic review of 536 ABM studies published between 2020 and 2023, applying standardized data extraction and nine original evaluation criteria—covering transparency, code sharing, model reusability, uncertainty quantification, stakeholder engagement, and multi-level validation. Contribution/Results: This is the first large-scale empirical assessment revealing critical credibility gaps: only 40.86% shared source code, 36.38% supported model reuse, 13.62% engaged stakeholders, and a mere 2.24% implemented comprehensive validation frameworks. The study identifies systemic weaknesses undermining ABM reliability and policy integration, and proposes actionable, evidence-based pathways to enhance ABM trustworthiness—including standardized reporting, open-code mandates, participatory modeling protocols, and tiered validation standards. These findings establish a methodological foundation and practical framework for rigorous, policy-ready computational modeling in future public health emergencies.

Technology Category

• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Agent-Based Simulation and Emergent Behavior
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

The COVID-19 pandemic prompted a surge in computational models to simulate disease dynamics and guide interventions. Agent-based models (ABMs) are well-suited to capture population and environmental heterogeneity, but their rapid deployment raised questions about utility for health policy. We systematically reviewed 536 COVID-19 ABM studies published from January 2020 to December 2023, retrieved from Web of Science, PubMed, and Wiley on January 30, 2024. Studies were included if they used ABMs to simulate COVID-19 transmission, where reviews were excluded. Studies were assessed against nine criteria of model usefulness, including transparency and re-use, interdisciplinary collaboration and stakeholder engagement, and evaluation practices. Publications peaked in late 2021 and were concentrated in a few countries. Most models explored behavioral or policy interventions (n = 294, 54.85%) rather than real-time forecasting (n = 9, 1.68%). While most described model assumptions (n = 491, 91.60%), fewer disclosed limitations (n = 349, 65.11%), shared code (n = 219, 40.86%), or built on existing models (n = 195, 36.38%). Standardized reporting protocols (n = 36, 6.72%) and stakeholder engagement were rare (13.62%, n = 73). Only 2.24% (n = 12) described a comprehensive validation framework, though uncertainty was often quantified (n = 407, 75.93%). Limitations of this review include underrepresentation of non-English studies, subjective data extraction, variability in study quality, and limited generalizability. Overall, COVID-19 ABMs advanced quickly, but lacked transparency, accessibility, and participatory engagement. Stronger standards are needed for ABMs to serve as reliable decision-support tools in future public health crises.

Problem

• Evaluating utility of agent-based models for health policy during COVID-19
• Assessing transparency and reusability of pandemic simulation models
• Identifying gaps in validation and stakeholder engagement practices

Innovation

• Systematic review of 536 agent-based models
• Assessed model transparency and re-use standards
• Evaluated interdisciplinary collaboration practices