SepsisAI Orchestrator: A Containerized and Scalable Platform for Deploying AI Models and Real-Time Monitoring in Early Sepsis Detection

2026-05-21

AI Summary

This study addresses key barriers to deploying clinical AI models—namely data heterogeneity, lack of deployment standardization, and stringent hospital requirements for high concurrency and low latency. The authors propose an open-source, modular platform integrating FHIR-inspired CDA preprocessing, NoSQL storage, containerized LightGBM model serving, and a Streamlit-based clinical dashboard, with scalable orchestration via Docker and Kubernetes. For the first time, they quantitatively demonstrate a U-shaped scaling behavior under clinical AI inference workloads: on a 12-thread CPU, increasing service replicas from 3 to 12 reduces p95 latency by 57.3% (from 3.3 s to 1.41 s) and eliminates request failures, whereas further scaling degrades performance due to scheduling contention. This work delivers an end-to-end, reproducible architecture for high-performance clinical AI deployment.

Abstract

Despite strong predictive results in the clinical machine learning literature, the translation of these models into bedside use remains limited by systems-level barriers: heterogeneous data representations, the absence of standardized deployment workflows, and a mismatch between research prototypes and the concurrency and latency requirements of hospital environments. We present the SepsisAI-Orchestrator, an open-source modular platform that addresses this deployment gap for early sepsis detection. The platform integrates HL7 FHIR-inspired Clinical Document Architecture (CDA) preprocessing, NoSQL storage, a containerized LightGBM classifier served via REST APIs, and a Streamlit clinical dashboard, orchestrated with Docker and Kubernetes. A previously validated LightGBM model (F1 0.87-0.94 on PhysioNet 2019) is reused without modification; the contribution lies in the surrounding infrastructure and its empirical characterization under load. Using k6 with 50-1000 concurrent virtual users, we find that replica count must be matched to the physical CPU thread count of the host: scaling from 3 to 12 replicas on a 12-thread CPU reduces p95 latency from 3.3s to 1.41s (57.3% reduction) and eliminates all request failures, while over-provisioning to 24 or 48 replicas degrades performance due to scheduler contention. To our knowledge this U-shaped scaling behavior has not been quantified previously for clinical AI inference workloads. We do not claim prospective clinical validation. Source code and deployment manifests are available at https://github.com/nucleusai/sepsisai-orchestrator.

Problem

• sepsis detection
• AI deployment
• clinical machine learning
• real-time monitoring
• healthcare systems

Innovation

• containerized deployment
• clinical AI infrastructure
• real-time sepsis detection
• scalability characterization
• FHIR-inspired preprocessing