ShuntServe: Cost-Efficient LLM Serving on Heterogeneous Spot GPU Clusters

2026-06-16

AI Summary

This work addresses the challenges of frequent instance preemption and load imbalance in large language model (LLM) inference on heterogeneous Spot GPU clusters. The authors propose a novel paradigm that jointly optimizes deployment and fault tolerance by integrating a Roofline performance model with a dynamic programming optimizer to co-design node configuration, parallelism strategy, and model layer placement. To enhance robustness and efficiency, they introduce output-preserving request migration and a concurrent initialization mechanism leveraging shared tensor storage. Evaluated on Llama-3.1-70B and Qwen3-32B, the approach achieves 1.42× and 1.35× higher throughput, respectively, compared to state-of-the-art baselines, while improving cost efficiency by over 31% for both offline and online inference relative to on-demand instances.

Abstract

As large language model (LLM) services become widely adopted, the cost of GPU resources for serving these models in cloud environments has emerged as a critical concern. Spot instances offer up to 90% cost savings over on-demand instances, but their frequent interruptions and limited availability pose significant challenges for continuous LLM serving. GPU spot instances, in particular, exhibit lower and more volatile availability than CPU-based instances, making homogeneous clusters that depend on a single GPU type vulnerable to correlated failures. Heterogeneous clusters spanning multiple GPU types can address this by leveraging complementary availability patterns across diverse spot pools, yet existing LLM serving systems are designed for homogeneous environments and suffer from load imbalance when deployed on heterogeneous GPUs. This paper presents ShuntServe, a cost-efficient LLM serving system for heterogeneous spot GPU clusters. ShuntServe employs a roofline model-based analytical serving performance estimator and a dynamic programming-based model placement optimizer that jointly determines node configuration, parallelization strategy, and layer assignment to maximize throughput across heterogeneous GPUs. To enhance fault tolerance when using spot instances, ShuntServe combines output-preserving request migration with concurrent initialization via a shared tensor store, minimizing migration downtime by overlapping replacement node preparation with ongoing serving. Evaluation on Llama-3.1-70B and Qwen3-32B with a heterogeneous AWS cluster of L4, A10G, and L40S GPUs shows that ShuntServe achieves 1.42x and 1.35x higher throughput than state-of-the-art baselines and attains 31.9% and 31.2% cost efficiency improvements over on-demand instances for offline and online serving, respectively.

Problem

• LLM serving
• heterogeneous GPU clusters
• spot instances
• cost efficiency
• load imbalance

Innovation

• heterogeneous spot GPU clusters
• cost-efficient LLM serving
• roofline model-based optimization
• output-preserving request migration
• dynamic programming-based placement