machine learning infrastructure and platforms

To address memory reliability challenges in large-scale AI training on Kubernetes—including OOM kills, over-allocation, memory leaks, and ephemeral storage exhaustion—this paper proposes the first memory governance framework tailored for ML workloads. Methodologically, it introduces a GPU-aware memory quota policy that jointly constrains GPU memory and system memory; designs a dynamic, cgroup v2–based elastic reclaim mechanism for ephemeral storage; and integrates a Prometheus/Grafana observability stack with a custom Eviction Advisor. Experiments on real-world, thousand-GPU distributed training clusters demonstrate a 92% reduction in OOM incidents, a 37% increase in GPU memory utilization, and SLA compliance exceeding 99.5% for training jobs. The core contribution lies in unifying memory QoS enforcement, GPU–system memory coupling modeling, and elastic ephemeral storage management within Kubernetes’ native scheduling architecture—marking the first such holistic approach.

This work addresses the challenges of SLO violations and resource inefficiency in machine learning model serving caused by inadequate capacity planning. To this end, the authors propose an adaptive, feedback-driven load testing framework that formalizes the ML serving load testing process for the first time. The framework incorporates real-traffic-based workload calibration and a warm-up mechanism, combined with adaptive search, performance signal feedback control, convergence detection, and GPU monitoring to efficiently estimate the maximum sustainable throughput under SLO constraints. Evaluation across 14 industrial cases demonstrates that the approach reduces capacity estimation error from approximately 30% to 2–6%, with the warm-up mechanism improving accuracy by 22.2%. This significantly mitigates deployment incidents and enhances GPU resource utilization efficiency.

Modern large-scale distributed training faces sharply diminishing returns in hardware scaling: as GPU counts reach thousands, communication overhead dominates performance bottlenecks, rendering conventional parallelism strategies—data, tensor, and pipeline parallelism—suboptimal. Method: Leveraging real-world LLM training workloads, this project establishes an empirical analytical framework spanning diverse model scales, hardware configurations, and parallelization strategies. It quantifies the nonlinear relationship between accelerator count and performance gain, precisely identifying critical inflection points across model, data, and compute scaling dimensions. Contribution/Results: We discover that low-communication “suboptimal” strategies become optimal at extreme scale; we empirically determine hardware selection criteria, cluster topology requirements, and optimal parallelism combinations for training billion-parameter models. Our findings provide actionable, deployment-ready optimization guidelines for trillion-parameter LLM training infrastructures.

This study addresses the GPU computational efficiency bottleneck in deep and machine learning. Methodologically, it proposes a task-aware GPU parallel architecture adaptation framework that systematically integrates CUDA stream-based concurrency, dynamic parallelism, and heterogeneous hardware (FPGA/TPU/ASIC) co-selection—implemented via deep integration into PyTorch, TensorFlow, and XGBoost. Its core contribution lies in establishing a transferable GPU optimization methodology, transcending model- or library-specific tuning. Experimental evaluation demonstrates 3–8× speedup across representative training and inference workloads. Furthermore, the authors open-source a modular, well-documented GPU optimization practice guide, substantially lowering the barrier to entry for AI practitioners seeking parallelization optimizations.

In deep learning training, repetitive hyperparameter tuning and neural architecture search cause frequent redundant data loading, leading to CPU bottlenecks, low GPU utilization, and resource waste. To address this, we propose the first cross-process shared data loading framework leveraging GPU-to-GPU direct interconnects (NVLink/PCIe RDMA), enabling multi-task reuse of a single preprocessing pipeline. Our approach is zero-intrusive—requiring no model code modification—and hardware- and framework-agnostic. By employing memory-shared mapping and a lightweight coordination protocol, it eliminates duplicate preprocessing operations. Experiments demonstrate up to 100% higher training throughput, a 50% reduction in cloud instance CPU resource demand, and corresponding 50% cost savings. Our method consistently outperforms state-of-the-art systems including CoorDL and Joader.

This study addresses the inequities in AI accessibility arising from imbalanced computational resource allocation, high energy consumption, and hardware barriers in large-scale deep learning training. Conducting systematic benchmarking on Intel Xeon CPUs and NVIDIA Tesla T4 GPUs across TensorFlow and PyTorch frameworks, we evaluate four representative models—Conv6, VGG16, ResNet18, and CycleGAN. For the first time, we jointly analyze model complexity, framework-level optimizations, and hardware platforms to quantify GPU acceleration benefits, revealing that lightweight models like Conv6 achieve the most significant speedup (up to 246× in training). We also find that TensorFlow’s operator fusion reduces inference latency by approximately 15%. Using polynomial regression, we project GPU memory requirements through 2025 and argue that shared GPU infrastructure is a critical pathway toward democratizing AI and ensuring equitable research opportunities.

This work addresses the significant communication bottleneck in multi-GPU training caused by the serial execution of computation and communication. The authors propose a portable runtime mechanism that requires no modifications to vendor libraries or kernels. By dynamically controlling on-chip resource occupancy of compute kernels, elevating the scheduling priority of communication streams, and leveraging shared memory for compute footprint management and cross-GPU resource coordination, the approach effectively enables concurrent execution of computation and collective communication. Evaluated on NVIDIA A40, A100, H100, and AMD MI250X GPUs, the method reduces end-to-end training time by up to 25.5%.

This work proposes ExecuTorch, the first end-to-end deployment framework natively integrated with the PyTorch ecosystem, addressing the fragmentation commonly encountered in edge AI deployment. By introducing an extensible backend abstraction, quantization-aware optimizations, and a unified model serialization format, ExecuTorch preserves the original model semantics while seamlessly targeting heterogeneous hardware—from microcontrollers to specialized accelerators—without sacrificing low latency or offline execution capabilities. The framework bridges the gap between research and production workflows, enabling consistent development and efficient deployment across a broad spectrum of devices, ranging from wearables to compute clusters, thereby significantly enhancing both deployment efficiency and cross-platform consistency.

The absence of a unified standard hinders observability, reproducibility, and hardware-software co-optimization of distributed machine learning workloads. This work proposes Chakra Execution Trace (ET), the first standardized, interoperable graph-based format tailored for distributed AI systems, which precisely captures critical operations, their dependencies, and resource constraints. An accompanying toolchain enables trace collection, analysis, synthesis, and replay, facilitating cross-platform performance benchmarking and co-design. The system has been validated on real-world AI clusters, adopted by MLCommons, and is being collaboratively developed by leading industry organizations including NVIDIA, AMD, and Meta.

This study presents the first systematic evaluation of the deployment feasibility of spiking neural networks (SNNs) in containerized edge environments. Focusing on resource- and energy-constrained virtual edge scenarios, we construct a testbed leveraging Docker Desktop, WSL2, and Windows 11 atop a single-node Kubernetes cluster orchestrated via K3d. We investigate end-to-end latency, throughput, classification accuracy, and concurrent behavior of SNN workloads under resource constraints and autoscaling conditions. Our findings reveal that SNNs are highly sensitive to CPU and memory availability: resource limitations substantially increase latency and reduce throughput, while classification accuracy remains stable. Furthermore, the default round-robin load-balancing strategy proves mismatched with SNNs’ long-duration inference tasks, leading to elevated tail latency. This work highlights the limitations of current stateless orchestration mechanisms in supporting neuromorphic computing paradigms.