Deployment of AI-Assisted Interventions: Capacity Constraints and Noisy Compliance

2026-04-15

AI Summary

This work addresses the suboptimality of conventional approaches that rely solely on prediction accuracy to set intervention thresholds and select algorithms in real-world settings characterized by constrained service capacity and stochastic individual compliance. To overcome this limitation, we propose a decision framework for AI-assisted intervention deployment that jointly optimizes the choice of predictive algorithm and intervention threshold, balancing resource utilization with coverage of high-value individuals. We formalize the shortcomings of standard strategies, introduce Operational AUC (OpAUC)—a novel evaluation metric aligned with operational objectives—and rigorously characterize the theoretical properties of optimal thresholds. Empirical evaluation on early sepsis detection data demonstrates that our approach significantly improves intervention effectiveness under limited resources.

Technology Category

• Search and Optimization: Algorithm Configuration
• Reasoning under Uncertainty: Stochastic Optimization
• Humans and AI: Planning and Decision Support for Human-Machine Teams

Application Category

• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success

Abstract

AI tools increasingly guide targeted interventions in healthcare, education, and recruiting. Algorithms score individuals, trigger outreach to those above a threshold (e.g., high-risk or high-value), and encourage them to request service; then providers deliver service to those who request. Standard practice sets the threshold and selects the algorithm to maximize predictive accuracy, assuming that better predictions yield better outcomes. We show that this approach is suboptimal when limited service capacity and probabilistic behavioral responses influence who receives service. In such settings, the optimal score threshold must balance two effects: ensuring all capacity is filled (utilization) and ensuring high-value individuals are served despite competition between requests (cannibalization). We characterize the optimal threshold and prove that policies based solely on predictive accuracy are generally suboptimal. Further, because optimal thresholds vary with service capacity, algorithm selection metrics like AUC, which weight all thresholds equally, are misaligned with operational performance. We introduce a new metric--Operational AUC (OpAUC)--and show it leads to optimal algorithm selection. Finally, we conduct a case study on sepsis early warning data and illustrate the magnitude of improvement that can be achieved from improved threshold and algorithm selection.

Problem

• AI-assisted interventions
• capacity constraints
• noisy compliance
• threshold selection
• algorithm selection

Innovation

• Operational AUC
• capacity constraints
• noisy compliance
• threshold optimization
• algorithm selection