Can A Society of Generative Agents Simulate Human Behavior and Inform Public Health Policy? A Case Study on Vaccine Hesitancy

2025-03-12

AI Summary

This study investigates the use of generative agents as substitutes for human participants in evaluating public health policies, using vaccine hesitancy as a case study. We propose VacSim, a simulation framework that constructs a virtual society of 100 LLM-driven agents, integrating demographic profiling, social network topology, and attitude evolution modeling to simulate vaccine decision-making and assess policy interventions. To enhance ecological validity, we introduce two novel mechanisms: simulation warm-up and attitude modulation. Additionally, we establish a multidimensional evaluation framework to assess LLM behavioral reliability. Experiments conducted with models including Llama and Qwen demonstrate that LLM agents partially reproduce empirically observed vaccine decision patterns, though systematic biases in demographic consistency persist. The framework delivers a reproducible, benchmarked pipeline for policy simulation. This work pioneers a generative-agent paradigm for public health policy evaluation, advancing computational epidemiology and AI-augmented policy analysis.

Technology Category

• Multiagent Systems: Agent-Based Simulation and Emergent Behavior
• Search and Optimization: Sampling/Simulation-based Search
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Can we simulate a sandbox society with generative agents to model human behavior, thereby reducing the over-reliance on real human trials for assessing public policies? In this work, we investigate the feasibility of simulating health-related decision-making, using vaccine hesitancy, defined as the delay in acceptance or refusal of vaccines despite the availability of vaccination services (MacDonald, 2015), as a case study. To this end, we introduce the VacSim framework with 100 generative agents powered by Large Language Models (LLMs). VacSim simulates vaccine policy outcomes with the following steps: 1) instantiate a population of agents with demographics based on census data; 2) connect the agents via a social network and model vaccine attitudes as a function of social dynamics and disease-related information; 3) design and evaluate various public health interventions aimed at mitigating vaccine hesitancy. To align with real-world results, we also introduce simulation warmup and attitude modulation to adjust agents' attitudes. We propose a series of evaluations to assess the reliability of various LLM simulations. Experiments indicate that models like Llama and Qwen can simulate aspects of human behavior but also highlight real-world alignment challenges, such as inconsistent responses with demographic profiles. This early exploration of LLM-driven simulations is not meant to serve as definitive policy guidance; instead, it serves as a call for action to examine social simulation for policy development.

Problem

• Simulate human behavior using generative agents for public health policy.
• Assess vaccine hesitancy through social dynamics and information modeling.
• Evaluate reliability of LLM simulations for real-world policy alignment.

Innovation

• Simulate society using generative agents
• Model vaccine attitudes via social networks
• Evaluate public health interventions effectively