ElastiCo: Elastic Configuration and Interference-Aware Orchestration for GPU Clusters

2026-08-08

AI Summary

This study addresses the low GPU utilization—approximately 25%—in GPU clusters caused by static allocation of training and inference workloads. To overcome this limitation, the authors propose an elastic colocation framework that, for the first time, integrates resource shape shifting, elastic shadow pricing, and a hardware counter–driven interference prediction model. This approach enables safe and dynamic sharing of training and inference tasks without requiring any modifications to user code. Implemented as a Kubernetes-native middleware, the framework reduces average job completion time to 1/2.94 of the baseline and improves cluster throughput by 2.02×, while raising GPU utilization to 46%, as demonstrated in both a 64-GPU physical testbed and a 512-GPU simulation environment.

Abstract

Modern GPU clusters must simultaneously serve deep learning training and offline large language model inference workloads, yet existing schedulers treat these as isolated resource consumers with rigid, static allocations. This leaves substantial GPU capacity underutilized: training jobs reserve entire devices despite periodic idle phases, while offline inference tasks over-provision GPUs despite bursty demand patterns. We present ElastiCo, an elastic co-location framework that enables training and inference workloads to safely share GPUs through three integrated mechanisms. First, Resource Shape Transformation exposes each job as a family of feasible resource-performance configurations. Second, Elastic Shadow Pricing decomposes the resulting multi-resource allocation problem into per-job configuration selection subproblems via dynamic per-resource shadow prices. Third, Interference-Aware Co-location uses a predictor trained on hardware-counter and task-level features to estimate pairwise performance degradation under GPU sharing. Implemented as native Kubernetes middleware requiring no user-code modifications, ElastiCo is evaluated on a 64-GPU testbed and through large-scale trace-driven simulations (up to 512 GPUs), reducing the average JCT by up to 2.94x, increasing the cluster throughput by 2.02x, and increasing the GPU utilization from approximately 25% to 46%.

Problem

• GPU clusters
• resource underutilization
• elastic configuration
• workload co-location
• interference-aware scheduling

Innovation

• elastic scheduling
• GPU sharing
• interference-aware co-location
• resource shape transformation
• shadow pricing