BiJuTy: An Interactive HPC-Aware Big Data Cluster Lifecycle Manager and Performance Assessment Utility for JupyterHub

2026-06-23

AI Summary

This work addresses the high barrier faced by novice users in deploying and optimizing big data workflows on high-performance computing (HPC) systems. To this end, it proposes an interactive JupyterHub-integrated plugin that, for the first time, unifies HPC-aware big data cluster lifecycle management and performance evaluation within a notebook environment. The approach seamlessly integrates resource scheduling, multi-cluster orchestration, performance metric collection, and visualization, enabling users to perform iterative optimization through an intuitive, one-click interface. Experimental results demonstrate that users can accomplish application deployment, performance assessment, and tuning with only a few clicks, substantially enhancing both usability and execution efficiency of big data workflows on HPC platforms.

Abstract

The increasing demand for data processing has created a pressing need for access to high-performance computing (HPC) systems. Nevertheless, leveraging these systems to execute complex big data processing workflows remains a significant challenge, especially for beginners. This work presents BiJuTy, a solution designed to bridge the accessibility gap for big data workflows on HPC systems within the Jupyter ecosystem. By providing an interactive and user-friendly interface, BiJuTy simplifies cluster lifecycle management and performance assessment, making it more accessible on HPC systems to beginners and experienced users alike. The solution is presented as an interactive interface that guides the user through the entire process, from setting up the cluster configuration to carrying out initial performance assessments. Additionally, the framework enables seamless management of multiple clusters directly within the Jupyter Notebook interface, eliminating the need to switch outside of working environment. The collection of performance metrics from various sources further simplifies the optimization workflow. Furthermore, an illustrative example is provided to demonstrate how BiJuTy can be deployed to optimize the performance of a big data processing application. This example showcases how the entire big data processing lifecycle can be iteratively executed and optimized in just a few clicks, helping to reach the goal of optimization easily and interactively. By facilitating such workflows, this work contributes in bringing the field of big data computing and high-performance computing one step closer to the goal of seamless interaction and usability.

Problem

• big data
• high-performance computing
• cluster lifecycle management
• performance assessment
• accessibility

Innovation

• HPC-aware
• cluster lifecycle management
• performance assessment
• JupyterHub integration
• interactive big data workflow