SwarmX: Agentic Scheduling for Low-Latency Agentic Systems

2026-06-19

AI Summary

Existing scheduling approaches struggle to handle the dynamic inference latencies and invocation patterns in agent-based AI applications induced by prompt semantics, resulting in high tail latency and low service efficiency. This work proposes the first scheduling framework tailored for agent AI systems, introducing a neural predictor that jointly models features from prompts, devices, runtime states, and models to enable tail-latency-aware routing and elastic scaling decisions. The framework employs a scheduler-agent co-design architecture to achieve online adaptability and seamless integration. Evaluated in production environments with thousands of GPUs and millions of CPU cores, the system reduces tail latency by up to 61.5% compared to state-of-the-art schedulers and achieves up to 2× higher throughput under the same service-level objective (SLO).

Abstract

Agentic AI applications compose multiple model calls and tool executions, creating new scheduling challenges for GPU-CPU clusters. Their inference time and model-call structure often depend on prompt semantics, making conventional scheduling approaches ineffective for low-latency serving. This paper presents SwarmX, a system that implements agentic scheduling for low-latency agentic applications. SwarmX uses scheduling-specific neural predictors to capture prompt, device, runtime, and target-model features; exposes distributional predictions to routers and scalers for tail-aware decisions; and provides mechanisms for predictor training and online adaptation. These predictors and mechanisms are integrated into a scheduler-agent framework that provides a common substrate for integration with existing scheduling and model-serving infrastructure. We evaluate SwarmX using production deployment (nearly one thousand GPUs and one million CPU cores) and controlled experiments on a 128-GPU testbed. Across multi-agent code generation, deep research, and multimodal agentic workflows, SwarmX reduces tail latency by up to 61.5% compared to state-of-the-art schedulers and sustains up to 2x the throughput of production schedulers under the same SLO.

Problem

• Agentic AI
• Low-Latency Scheduling
• GPU-CPU Clusters
• Tail Latency
• Model Serving

Innovation

• agentic scheduling
• neural predictors
• tail-latency optimization
• online adaptation
• GPU-CPU cluster