An AI-Based Public Health Data Monitoring System

2025-06-04

AI Summary

Public health data volumes are surging, rendering conventional threshold-based alerting systems inadequate due to delayed responses, frequent parameter recalibration, and poor scalability. To address these challenges, this paper proposes an AI-driven real-time monitoring paradigm for large-scale temporal health data (e.g., case counts, hospitalizations, deaths), shifting from binary alerts to ranked anomaly prioritization—integrating human-centered design principles with adaptive anomaly detection. We develop a multimodal temporal anomaly detection model featuring dynamic weight learning and deploy it via a lightweight architecture capable of processing 5 million data points daily with low latency. The system has been deployed at a national public health agency, where empirical evaluation demonstrates a 54× improvement in surveillance review efficiency. It significantly enhances early outbreak detection and data quality diagnostics, establishing a novel methodology and technical framework for real-time, scalable, and human-in-the-loop intelligent public health monitoring.

Technology Category

• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Humans and AI: Human-in-the-loop Machine Learning
• Machine Learning: Hardware-aware ML

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Public health experts need scalable approaches to monitor large volumes of health data (e.g., cases, hospitalizations, deaths) for outbreaks or data quality issues. Traditional alert-based monitoring systems struggle with modern public health data monitoring systems for several reasons, including that alerting thresholds need to be constantly reset and the data volumes may cause application lag. Instead, we propose a ranking-based monitoring paradigm that leverages new AI anomaly detection methods. Through a multi-year interdisciplinary collaboration, the resulting system has been deployed at a national organization to monitor up to 5,000,000 data points daily. A three-month longitudinal deployed evaluation revealed a significant improvement in monitoring objectives, with a 54x increase in reviewer speed efficiency compared to traditional alert-based methods. This work highlights the potential of human-centered AI to transform public health decision-making.

Problem

• Monitoring large-scale public health data for outbreaks and quality issues
• Overcoming limitations of traditional alert-based monitoring systems
• Improving efficiency in public health data review using AI anomaly detection

Innovation

• AI anomaly detection for health data
• Ranking-based monitoring paradigm
• Scalable daily data point analysis