Energy-Aware Scheduling for Serverless LLM Serving on Shared GPUs

2026-06-29

AI Summary

This work addresses the challenge of jointly optimizing energy consumption and latency when co-locating multiple large language models (LLMs) on shared GPUs for serverless inference. To this end, the authors propose Festina, a system that prioritizes cluster-wide energy minimization while meeting time-to-first-token (TTFT) and time-between-tokens (TBT) service level objectives (SLOs). Festina achieves this through coordinated scheduling of request placement, streaming multiprocessor (SM) partitioning, and GPU dynamic voltage and frequency scaling (DVFS), leveraging both offline profiling and online state summarization to enable fast global decisions and phase-aware local resource adjustments. Additionally, it introduces SLO-aware request batching to reduce static power consumption. Experimental results demonstrate that Festina reduces energy consumption by up to 56% compared to state-of-the-art systems, while maintaining SLO violation rates below 2%.

Abstract

As LLM inference becomes a major cloud workload, its growing energy footprint makes cluster-wide energy optimization increasingly important. Serverless LLM serving helps platforms absorb traffic volatility by elastically sharing GPU resources across models, but this sharing also makes energy optimization difficult. Multiple co-resident models run under one device-wide operating point, while their resource demands and latency slack change across execution phases and load conditions. As a result, minimizing energy requires coordinated scheduling across request placement, runtime resource adaptation, and workload consolidation. We present Festina, a profiling-guided, power-aware control plane to minimize cluster-wide energy for serverless LLM serving. Unlike common global-local schedulers that focus on throughput or tail latency, Festina makes energy-first decisions by jointly coordinating request placement, SM partitioning, and GPU operating points under TTFT/TBT SLOs. In our system, a lightweight global scheduler performs fast, SLO-safe, energy-aware placement using constant-time lookups from offline profiles and GPU state summaries. On each GPU, a phase-aware local scheduler continuously adapts task batching and compute resources to minimize power consumption. Festina further performs energy-aware workload consolidation to reduce GPUs' static power consumption via SLO-aware migration. Comparison with four SOTA LLM serving systems and one DVFS-augmented system demonstrates that Festina reduces energy consumption by up to 56% while maintaining parity in SLO attainment (within a 2% margin)

Problem

• Energy-Aware Scheduling
• Serverless LLM Serving
• Shared GPUs
• Energy Optimization
• Latency SLOs

Innovation

• energy-aware scheduling
• serverless LLM serving
• GPU resource sharing
• SLO-aware consolidation
• power optimization