Pythia: Toward Predictability-Driven Agent-Native LLM Serving

2026-04-28

AI Summary

This work addresses the inefficiencies in current LLM serving systems, which treat multi-agent workloads as generic traffic and overlook the structured semantics that enable predictable execution patterns. This oversight leads to low cache hit rates, contention over long-context resources, and high scheduling latency. To overcome these limitations, the paper introduces a lightweight interface at the serving layer that explicitly captures multi-agent workflow semantics, enabling an agent-native LLM serving system. It proposes a novel predictability-driven optimization paradigm, incorporating workflow-aware request scheduling, prefix caching, dynamic resource allocation, and elastic scaling. Evaluated in real-world production environments, the approach significantly improves throughput and substantially reduces job completion time, demonstrating both efficiency and practicality.

Abstract

As LLM applications grow more complex, developers are increasingly adopting multi-agent architectures to decompose workflows into specialized, collaborative components, introducing structure that constrains agent behavior and exposes useful semantic predictability. Unlike traditional LLM serving, which operates under highly dynamic and uncertain conditions, this structured topology enables opportunities to reduce runtime uncertainty -- yet existing systems fail to exploit it, treating agentic workloads as generic traffic and incurring significant inefficiencies. Our analysis of production traces from an agent-serving platform and an internal coding assistant reveals key bottlenecks, including low prefix cache hit rates, severe resource contention from long-context requests, and substantial queuing delays due to suboptimal scaling. To address these challenges, we propose Pythia, a multi-agent serving system that captures workflow semantics through a simple interface at the serving layer, unlocking new optimization opportunities and substantially improving throughput and job completion time over state-of-the-art baselines.

Problem

• multi-agent LLM serving
• semantic predictability
• runtime uncertainty
• resource contention
• queuing delay

Innovation

• predictability-driven
• agent-native
• LLM serving
• workflow semantics
• prefix caching