LLM-based Agent Simulation for Maternal Health Interventions: Uncertainty Estimation and Decision-focused Evaluation

2025-03-25

AI Summary

In primary maternal and child health care, data scarcity impedes rigorous evaluation of intervention effectiveness. Method: This paper proposes an LLM-based agent simulation framework to predict pregnant women’s reception behavior toward two health communication modalities—automated SMS (control) versus human-led follow-up (intervention)—while quantifying predictive uncertainty. It introduces a novel LLM-driven binary entropy uncertainty estimation, integrated with multi-sample ensembling and model calibration, to establish a decision-oriented evaluation paradigm wherein probabilistic outputs directly inform intervention feasibility assessment and trial design decisions. Results: Experiments demonstrate significant improvements in F1 score and probability calibration. The framework enables real-world mHealth deployment even in the absence of historical control data and generalizes to broader public health and emergency response applications.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Applications
• Multiagent Systems: Agent Communication

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Agent-based simulation is crucial for modeling complex human behavior, yet traditional approaches require extensive domain knowledge and large datasets. In data-scarce healthcare settings where historic and counterfactual data are limited, large language models (LLMs) offer a promising alternative by leveraging broad world knowledge. This study examines an LLM-driven simulation of a maternal mobile health program, predicting beneficiaries' listening behavior when they receive health information via automated messages (control) or live representatives (intervention). Since uncertainty quantification is critical for decision-making in health interventions, we propose an LLM epistemic uncertainty estimation method based on binary entropy across multiple samples. We enhance model robustness through ensemble approaches, improving F1 score and model calibration compared to individual models. Beyond direct evaluation, we take a decision-focused approach, demonstrating how LLM predictions inform intervention feasibility and trial implementation in data-limited settings. The proposed method extends to public health, disaster response, and other domains requiring rapid intervention assessment under severe data constraints. All code and prompts used for this work can be found at https://github.com/sarahmart/LLM-ABS-ARMMAN-prediction.

Problem

• Simulating maternal health interventions using LLMs with limited data
• Estimating uncertainty in LLM predictions for healthcare decision-making
• Enhancing model robustness for intervention feasibility in data-scarce settings

Innovation

• LLM-driven simulation for maternal health
• Epistemic uncertainty estimation via binary entropy
• Ensemble approaches enhance model robustness