🤖 AI Summary
To address memory reliability challenges in large-scale AI training on Kubernetes—including OOM kills, over-allocation, memory leaks, and ephemeral storage exhaustion—this paper proposes the first memory governance framework tailored for ML workloads. Methodologically, it introduces a GPU-aware memory quota policy that jointly constrains GPU memory and system memory; designs a dynamic, cgroup v2–based elastic reclaim mechanism for ephemeral storage; and integrates a Prometheus/Grafana observability stack with a custom Eviction Advisor. Experiments on real-world, thousand-GPU distributed training clusters demonstrate a 92% reduction in OOM incidents, a 37% increase in GPU memory utilization, and SLA compliance exceeding 99.5% for training jobs. The core contribution lies in unifying memory QoS enforcement, GPU–system memory coupling modeling, and elastic ephemeral storage management within Kubernetes’ native scheduling architecture—marking the first such holistic approach.
📝 Abstract
Kubernetes offers a powerful orchestration platform for machine learning training, but memory management can be challenging due to specialized needs and resource constraints. This paper outlines how Kubernetes handles memory requests, limits, Quality of Service classes, and eviction policies for ML workloads, with special focus on GPU memory and ephemeral storage. Common pitfalls such as overcommitment, memory leaks, and ephemeral volume exhaustion are examined. We then provide best practices for stable, scalable memory utilization to help ML practitioners prevent out-of-memory events and ensure high-performance ML training pipelines.