CoCoScale: Leveraging Layer-wise Scaling to Unlock the Potential of Online LLM Serving

2026-07-05

AI Summary

This work addresses the inefficiencies of conventional instance-level scaling in online large language model (LLM) services, which often suffer from sluggish response and suboptimal resource utilization under dynamically imbalanced workloads. The paper introduces, for the first time, a layer-granularity dynamic scaling mechanism that elastically increases the parallelism of frequently accessed model layers by leveraging otherwise idle device resources. This approach enables fine-grained elastic data parallelism without modifying the model architecture or incurring additional hardware overhead. By integrating hierarchical parallel scheduling with resource reclamation and reallocation strategies, the proposed method substantially outperforms traditional scaling techniques: it reduces cold-start latency by 97.9%–99.3%, decreases average request latency by 20.7%–28.1% under real-world production workloads, and consistently meets 100% of service-level objectives.

Abstract

Online large language model (LLM) serving has become the backbone of modern AI applications, powering diverse downstream services through shared hardware clusters. However, modern serving systems frequently encounter highly dynamic workloads characterized by severe workload skewness, where a small fraction of model instances receives the vast majority of traffic. Existing instance-level scaling mechanisms are limited by coarse-grained resource adjustment: scaling up requires the cold-start of full-model replicas, incurring substantial latency, while scaling down leaves the system vulnerable to performance degradation during sudden traffic surges. The key insight of this work is that LLM serving offers a unique opportunity for fine-grained scaling. In this paper, we propose CoCoScale, a layer-wise dynamic scaling mechanism that selectively expands the parallelism of hot layers onto idle resources reclaimed from underutilized devices, enabling elastic data parallelism without altering model architectures or adding hardware overhead. Evaluations demonstrate that CoCoScale significantly reduces cold start latency by 97.9%-99.3% compared to traditional scale up. Under production traces, CoCoScale reduces average latency by 20.7\%--28.1\% and achieves full Service Level Objective (SLO) attainment, demonstrating superior dynamic adaptability and resource efficiency.

Problem

• online LLM serving
• workload skewness
• instance-level scaling
• cold-start latency
• dynamic workloads

Innovation

• layer-wise scaling
• online LLM serving
• elastic data parallelism
• cold-start latency
• resource efficiency