Automated Dynamic AI Inference Scaling on HPC-Infrastructure: Integrating Kubernetes, Slurm and vLLM

2025-11-26

AI Summary

The surge in AI inference demands at academic institutions conflicts with traditional HPC schedulers’ inability to support dynamic, low-latency, interactive large language model (LLM) services. Method: We propose a heterogeneous scheduling architecture that synergistically integrates Kubernetes and Slurm, enabling automatic elastic scaling for dynamic AI workloads in supercomputing environments. Our approach unifies the vLLM high-throughput inference engine, Kubernetes container orchestration, and Slurm-based HPC resource management via a unified scheduling interface. Contribution/Results: Evaluated on the RAMSES supercomputing platform, the system supports thousands of concurrent requests with only ~500 ms end-to-end latency overhead, achieving both high resource utilization and sub-second responsiveness. This work establishes a scalable, low-latency infrastructure paradigm for delivering AI-as-a-Service (AIaaS) on HPC systems, bridging the gap between AI service requirements and legacy HPC scheduling capabilities.

Technology Category

• Machine Learning: Hardware-aware ML
• Planning, Routing, and Scheduling: Planning with Language Models
• Humans and AI: Other Foundations of Human Computation & AI

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Due to rising demands for Artificial Inteligence (AI) inference, especially in higher education, novel solutions utilising existing infrastructure are emerging. The utilisation of High-Performance Computing (HPC) has become a prevalent approach for the implementation of such solutions. However, the classical operating model of HPC does not adapt well to the requirements of synchronous, user-facing dynamic AI application workloads. In this paper, we propose our solution that serves LLMs by integrating vLLM, Slurm and Kubernetes on the supercomputer extit{RAMSES}. The initial benchmark indicates that the proposed architecture scales efficiently for 100, 500 and 1000 concurrent requests, incurring only an overhead of approximately 500 ms in terms of end-to-end latency.

Problem

• Dynamic AI inference scaling on HPC infrastructure
• Integrating Kubernetes Slurm vLLM for LLM serving
• Reducing latency overhead for concurrent AI requests

Innovation

• Integrating vLLM with Kubernetes and Slurm
• Dynamic AI inference scaling on HPC infrastructure
• Efficient scaling for concurrent requests with minimal latency