Introducing JIRIAF: A Virtual Kubelet Integration for Optimizing HPC Resource Provisioning

2025-02-25

AI Summary

To address challenges in HPC environments—including heterogeneous resource scheduling complexity, constraints on containerized application deployment, and lagging dynamic workload modeling—this paper proposes JIRIAF, a unified scheduling framework. Methodologically, JIRIAF introduces (i) the first HPC-oriented lightweight Virtual Kubelet adaptation architecture, enabling fine-grained, low-privilege container scheduling across heterogeneous infrastructures via tight coordination between Kubernetes and a user-space resource manager (JRM); and (ii) a streaming task queue modeling and closed-loop optimization mechanism integrating digital twin principles with dynamic Bayesian networks (DBNs). Evaluated on the NERSC Perlmutter system, JIRIAF achieves unified cross-facility orchestration, improves resource utilization by 23%, reduces job startup latency by 41%, and significantly enhances the elasticity of large-scale HPC applications in cloud-native environments.

Technology Category

• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems
• Machine Learning: Hardware-aware ML
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Virtualization and resource management in Web systems and infrastructures
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

The JIRIAF (JLab Integrated Research Infrastructure Across Facilities) framework is designed to streamline resource management and optimize high-performance computing (HPC) workloads across heterogeneous environments. Central to JIRIAF is the JIRIAF Resource Manager (JRM), which effectively leverages Kubernetes and Virtual Kubelet to manage resources dynamically, even in environments with restricted user privileges. By operating in userspace, JRM facilitates the execution of user applications as containers across diverse computing sites, ensuring unified control and monitoring. The framework's effectiveness is demonstrated through a case study involving the deployment of data-stream processing pipelines on the Perlmutter system at NERSC, showcasing its capability to manage large-scale HPC applications efficiently. Additionally, we discuss the integration of a digital twin model for a simulated queue system related to a streaming system, using a Dynamic Bayesian Network (DBN) to enhance real-time monitoring and control, providing valuable insights into system performance and optimization strategies.

Problem

• Optimizing HPC resource provisioning
• Dynamic management of heterogeneous environments
• Enhancing real-time monitoring with digital twin

Innovation

• Virtual Kubelet integration
• Dynamic resource management
• Digital twin simulation