SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling

2026-07-09

AI Summary

Existing LLM schedulers struggle to simultaneously achieve KV cache reuse and cluster load balancing in agent-serving scenarios, limiting throughput performance. This work proposes a session-aware balanced scheduling strategy that performs global load-balanced placement only for the first request of each session, while subsequent requests retain locality by adhering to the same node to exploit intra-session access patterns. The approach employs a stateless design relying solely on session-round information provided by users, integrating session-centric scheduling with a coordinated global-local KV storage mechanism. Evaluated under realistic agent workloads, the method improves cluster throughput by 10–16% in co-located prefill-decode architectures and boosts prefill throughput by 2–34% in disaggregated setups, while significantly reducing per-token latency.

Abstract

LLM scheduling is critical to serving, yet it remains unclear how well existing designs fit agentic serving--with LLM requests issued by agents instead of humans. This shifts the workload in two ways: (1) agents act only on complete responses, making the cluster's tokens per second (TPS) the primary goal and relaxing--not eliminating--per-token latency requirements; and (2) requests share much of their KV\$-reuse exceeds 80% of request tokens in a production trace from BAILIAN, versus 54-62% in chat. This paper first contributes a systematic study of request scheduling for agents on two real-world traces. We find that to increase KV\$ reuse, existing schedulers overly prioritize routing requests to instances caching their KV\$, overloading a few while leaving the rest idle, capping TPS. We thus present two key insights: (1) load balance need not sacrifice all KV\$ reuse, thanks to the global-tier KV\$ store and (2) by utilizing the workload's intra-session locality, balancing a small fraction of requests--the first request in each agent session--suffices to balance the cluster without sacrificing most KV\$ reuse on local instances. SMETRIC realizes these insights with balanced session-centric scheduling: it routes each session's first request purely for load balance and its follow-up requests in a cache-aware manner, preserving load balance and local reuse while keeping demand on the global tier low. Using the session turn information as the scheduling metric is deliberate: it is derived efficiently and accurately from the user inputs alone, so the scheduler stays clean and stateless. SMETRIC improves cluster TPS by 10-16% under prefill-decode colocation with a global store and prefill TPS by 2-34% under disaggregation over state-of-the-art schedulers, also with a better per-token latency.

Problem

• LLM scheduling
• agentic serving
• KV cache reuse
• load balancing
• tokens per second

Innovation

• session-centric scheduling
• KV cache reuse
• load balancing
• LLM serving
• agent workloads