OpScale: Operator-level Provisioning and Autoscaling for LLM Serving

2026-08-13

AI Summary

This work addresses the inefficiency of existing LLM serving systems, which scale entire models as monolithic units and thus struggle to handle bursty workloads, often leading to SLO violations or underutilized GPU resources. To overcome these limitations, the paper introduces the first operator-level elasticity framework that exploits heterogeneity and elasticity across individual model operators. By co-optimizing operator-level performance profiling, resource provisioning, deployment policies, and runtime scheduling, the proposed approach achieves fine-grained resource management beyond conventional model-level scaling. Experiments on clusters with 40 A100 and 24 GB200 GPUs demonstrate that the system reduces GPU usage by up to 36.3% and power consumption by 28% compared to baselines, or alternatively improves throughput by 44% under fixed hardware costs.

Abstract

Achieving cost efficiency while meeting strict user-facing SLOs (e.g., time-to-first-token) remains a fundamental challenge for cloud GPU clusters serving large language models (LLMs). Autoscaling is the key mechanism for cluster resource management, yet a basic system design question is open for serving LLMs: what should be the unit of scaling? Existing approaches primarily treat the entire model as a monolithic scaling unit--simple but unable to capture the fine-grained dynamics of inference workloads. As a result, such coarse-grained scaling often leads to either SLO violations under bursty demand or significant GPU under-utilization. Our characterization reveals substantial operator heterogeneity, exposing operator-level elasticity as a viable scaling primitive. We present OpScale, a practical operator-level orchestration framework of profiling, provisioning, placement, and runtime serving. OpScale is designed to tackle the high complexity and the space explosion problem, arising from operating at this finer granularity. Evaluated with production traces on up to 40 A100s and 24 GB200s, OpScale attains SLOs with up to 36.3% fewer GPUs and 28% less power, or achieves 44% higher throughput under fixed cost budgets.

Problem

• LLM serving
• autoscaling
• SLO
• GPU utilization
• operator-level elasticity

Innovation

• operator-level autoscaling
• LLM serving
• resource provisioning
• SLO-aware scaling
• GPU orchestration