Scalable HPC Job Scheduling and Resource Management in SST

2024-12-15

Online World Conference on Soft Computing in Industrial Applications

AI Summary

To address low scheduling accuracy for scientific and data-intensive workflows, and insufficient dynamic resource management in high-performance computing (HPC), this paper designs and implements the first scalable job scheduling and resource management component tailored for the Structural Simulation Toolkit (SST). The component innovatively integrates a workflow management module into SST—enabling task dependency modeling, dynamic resource allocation, and comparative simulation of multiple scheduling algorithms (e.g., Shortest Job First, Fair Share)—while incorporating resource reservation policies and a parallel event-processing mechanism. Experimental evaluation demonstrates sub-5% error in job waiting time and sub-3% error in node utilization across diverse workloads, alongside strong scalability to thousands of cores. This advancement significantly enhances the fidelity of cycle-accurate simulation in capturing real-world scheduler behavior.

Technology Category

• Planning, Routing, and Scheduling: Scheduling
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Virtualization and resource management in Web systems and infrastructures
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Efficient job scheduling and resource management contributes towards system throughput and efficiency maximization in high-performance computing (HPC) systems. In this paper, we introduce a scalable job scheduling and resource management component within the structural simulation toolkit (SST), a cycle-accurate and parallel discrete-event simulator. Our proposed simulator includes state-of-the-art job scheduling algorithms and resource management techniques. Additionally, it introduces a workflow management components that supports the simulation of task dependencies and resource allocations, crucial for workflows typical in scientific computing and data-intensive applications. We present validation and scalability results of our job scheduling simulator. Simulation shows that our simulator achieves good accuracy in various metrics (e.g., job wait times, number of nodes usage) and also achieves good parallel performance.

Problem

• High-Performance Computing
• Task Scheduling
• Resource Management

Innovation

• High-performance computing
• Advanced scheduling algorithms
• Resource management