Scepsy: Serving Agentic Workflows Using Aggregate LLM Pipelines

2026-04-16

AI Summary

This work addresses the challenge of simultaneously achieving low latency and high throughput in multi-agent LLM workflows, which suffer from structural complexity, unpredictable execution times, and GPU resource contention. The authors propose a lightweight performance prediction model based on aggregated execution shares, which jointly optimizes GPU allocation, tensor parallelism degree, and replica count. A hierarchical heuristic scheduling algorithm is further designed to minimize resource fragmentation and end-to-end latency while respecting network topology constraints. Experimental results demonstrate that the proposed approach improves system throughput by up to 2.4× and reduces latency by up to 27× compared to existing methods that rely on isolated optimizations or manual configuration.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Multiagent Systems: Teamwork

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Agentic workflows carry out complex tasks by orchestrating multiple large language models (LLMs) and tools. Serving such workflows at a target throughput with low latency is challenging because they can be defined using arbitrary agentic frameworks and exhibit unpredictable execution times: execution may branch, fan-out, or recur in data-dependent ways. Since LLMs in workflows often outnumber available GPUs, their execution also leads to GPU oversubscription. We describe Scepsy, a new agentic serving system that efficiently schedules arbitrary multi-LLM agentic workflows onto a GPU cluster. Scepsy exploits the insight that, while agentic workflows have unpredictable end-to-end latencies, the shares of each LLM's total execution times are comparatively stable across executions. Scepsy decides on GPU allocations based on these aggregate shares: first, it profiles the LLMs under different parallelism degrees. It then uses these statistics to construct an Aggregate LLM Pipeline, which is a lightweight latency/throughput predictor for allocations. To find a GPU allocation that minimizes latency while achieving a target throughput, Scepsy uses the Aggregate LLM Pipeline to explore a search space over fractional GPU shares, tensor parallelism degrees, and replica counts. It uses a hierarchical heuristic to place the best allocation onto the GPU cluster, minimizing fragmentation, while respecting network topology constraints. Our evaluation on realistic agentic workflows shows that Scepsy achieves up to 2.4x higher throughput and 27x lower latency compared to systems that optimize LLMs independently or rely on user-specified allocations.

Problem

• agentic workflows
• LLM serving
• GPU oversubscription
• latency
• throughput

Innovation

• Agentic Workflows
• LLM Serving
• GPU Scheduling
• Aggregate LLM Pipeline
• Resource Allocation