Context in Public Health for Underserved Communities: A Bayesian Approach to Online Restless Bandits

2024-02-07

AI Summary

This paper addresses the challenge of allocating public health interventions to resource-constrained, underserved communities by formulating it as an online restless multi-armed bandit (RMAB) problem with context-dependent transitions and non-stationary dynamics—aiming to rapidly learn individual health state transition patterns within short time horizons and limited intervention budgets to maximize cumulative health rewards. Method: We propose the first integration of Bayesian state inference with context-aware Thompson sampling, enabling both cross-individual and intra-individual information sharing to enhance policy adaptability and sample efficiency in small-data, fast-changing environments. Results: Experiments on real-world maternal mobile health data from ARMMAN in India demonstrate that our method significantly outperforms existing baselines in terms of cumulative reward under limited samples and practical deployability.

Technology Category

• Machine Learning: Online Learning & Bandits
• Intelligent Robots: State Estimation
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Public health programs often provide interventions to encourage program adherence, and effectively allocating interventions is vital for producing the greatest overall health outcomes, especially in underserved communities where resources are limited. Such resource allocation problems are often modeled as restless multi-armed bandits (RMABs) with unknown underlying transition dynamics, hence requiring online reinforcement learning (RL). We present Bayesian Learning for Contextual RMABs (BCoR), an online RL approach for RMABs that novelly combines techniques in Bayesian modeling with Thompson sampling to flexibly model the complex RMAB settings present in public health program adherence problems, namely context and non-stationarity. BCoR's key strength is the ability to leverage shared information within and between arms to learn the unknown RMAB transition dynamics quickly in intervention-scarce settings with relatively short time horizons, which is common in public health applications. Empirically, BCoR achieves substantially higher finite-sample performance over a range of experimental settings, including a setting using real-world adherence data that was developed in collaboration with ARMMAN, an NGO in India which runs a large-scale maternal mHealth program, showcasing BCoR practical utility and potential for real-world deployment.

Problem

• Optimizes public health resource allocation
• Addresses underserved communities' limited resources
• Enhances intervention effectiveness with Bayesian RL

Innovation

• Bayesian modeling with Thompson sampling
• Online reinforcement learning for RMABs
• Leverages shared information within arms