Improved Disease Outbreak Detection from Out-of-sequence measurements Using Markov-switching Fixed-lag Particle Filters

2025-12-01

AI Summary

In disease surveillance, delayed or out-of-sequence (OOS) observations hinder traditional particle filters from effectively backtracking and correcting historical states. To address this, we propose a Markov-switching fixed-lag particle filtering framework. Our method jointly refines latent state trajectories and estimates model parameters online by resampling historical particle trajectories, integrating fixed-lag smoothing with the SMC² algorithm. Key innovations include: (i) the first incorporation of a Markov-switching mechanism into fixed-lag particle smoothing, enabling simultaneous retrospective updates of both states and model parameters under OOS observations; and (ii) significant improvements in epidemic phase identification accuracy and outbreak detection timeliness—demonstrated on real-world delayed-data scenarios—with reduced false alarm rates, enhanced robustness, and adaptive responsiveness.

Technology Category

• Intelligent Robots: State Estimation
• Search and Optimization: Mixed Discrete/Continuous Search
• Machine Learning: Time-Series/Data Streams

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Particle filters (PFs) have become an essential tool for disease surveillance, as they can estimate hidden epidemic states in nonlinear and non-Gaussian models. In epidemic modelling, population dynamics may be governed by distinct regimes such as endemic or outbreak phases which can be represented using Markov-switching state-space models. In many real-world surveillance systems, data often arrives with delays or in the wrong temporal order, producing out-of-sequence (OOS) measurements that pertain to past time points rather than the current one. While existing PF methods can incorporate OOS measurements through particle reweighting, these approaches are limited in their ability to fully adjust past latent trajectories. To address this, we introduce a Markov-switching fixed-lag particle filter (FL-PF) that resimulates particle trajectories within a user-specified lag window, allowing OOS measurements to retroactively update both state and model estimates. By explicitly reevaluating historical samples, the FL-PF improves the accuracy and timeliness of outbreak detection and reduces false alarms. We also show how to compute the log-likelihood within the FL-PF framework, enabling parameter estimation using Sequential Monte Carlo squared (SMC$^2$). Together, these contributions extend the applicability of PFs to surveillance systems where retrospective data are common, offering a more robust framework for monitoring disease outbreaks and parameter inference.

Problem

• Detects disease outbreaks from delayed or misordered data
• Improves accuracy and reduces false alarms in surveillance
• Enables parameter estimation with retrospective data updates

Innovation

• Markov-switching fixed-lag particle filter for delayed data
• Resimulates particle trajectories within lag window
• Improves outbreak detection accuracy and reduces false alarms