🤖 AI Summary
This work addresses the inefficiencies of traditional rigid job scheduling in high-performance computing (HPC) clusters, which often result in low resource utilization and prolonged job waiting times. The authors propose a novel malleable job scheduling strategy that dynamically adjusts resource allocations at runtime while prioritizing each job’s preferred configuration. They systematically investigate the interplay among workload characteristics, the proportion of malleable jobs, and scheduling policies. Using the ElastiSim simulation framework and real-world workload traces from the Cori, Eagle, and Theta supercomputers, they evaluate five scheduling strategies across malleable job ratios ranging from 0% to 100%. Experimental results demonstrate that, compared to fully rigid scheduling, the best-performing strategy reduces job turnaround time by 37–67%, shortens makespan by 16–65%, decreases waiting time by 73–99%, and improves node utilization by 5–52%.
📝 Abstract
Optimizing resource utilization in high-performance computing (HPC) clusters is essential for maximizing both system efficiency and user satisfaction. However, traditional rigid job scheduling often results in underutilized resources and increased job waiting times.
This work evaluates the benefits of resource elasticity, where the job scheduler dynamically adjusts the resource allocation of malleable jobs at runtime. Using real workload traces from the Cori, Eagle, and Theta supercomputers, we simulate varying proportions (0-100%) of malleable jobs with the ElastiSim software.
We evaluate five job scheduling strategies, including a novel one that maintains malleable jobs at their preferred resource allocation when possible. Results show that, compared to fully rigid workloads, malleable jobs yield significant improvements across all key metrics. Considering the best-performing scheduling strategy for each supercomputer, job turnaround times decrease by 37-67%, job makespan by 16-65%, job wait times by 73-99%, and node utilization improves by 5-52%. Although improvements vary, gains remain substantial even at 20% malleable jobs.
This work highlights important correlations between workload characteristics (e.g., job runtimes and node requirements), malleability proportions, and scheduling strategies. These findings confirm the potential of malleability to address inefficiencies in current HPC practices and demonstrate that even limited adoption can provide substantial advantages, encouraging its integration into HPC resource management.