Sizey: Memory-Efficient Execution of Scientific Workflow Tasks

2024-07-23

IEEE International Conference on Cluster Computing

AI Summary

Scientific workflow tasks exhibit heterogeneous inputs and types, making memory demand prediction challenging—leading to resource over-allocation, reduced cluster throughput, and increased task failure risk. To address this, we propose an online dynamic memory prediction method featuring novel parallel online multi-model training and adaptive model selection. We introduce Resource Allocation Quality (RAQ) as a new metric for evaluating allocation efficacy and support continuous runtime retraining and real-time optimization. Our approach integrates ensemble machine learning, online learning, fine-grained resource monitoring, and workflow runtime analysis. Evaluated on six real-world nf-core workflows, our method reduces median memory waste by 24.68% compared to the state-of-the-art baseline, significantly improving resource utilization and system throughput.

Technology Category

• Machine Learning: Hardware-aware ML
• Search and Optimization: Metareasoning and Metaheuristics
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

As the amount of available data continues to grow in fields as diverse as bioinformatics, physics, and remote sensing, the importance of scientific workflows in the design and im-plementation of reproducible data analysis pipelines increases. When developing workflows, resource requirements must be defined for each type of task in the workflow. Typically, task types vary widely in their computational demands because they are simply wrappers for arbitrary black-box analysis tools. Furthermore, the resource consumption for the same task type can vary considerably as well due to different inputs. Since underestimating memory resources leads to bottlenecks and task failures, workflow developers tend to overestimate memory resources. However, overprovisioning of memory wastes resources and limits cluster throughout. Addressing this problem, we propose Sizey, a novel online memory prediction method for workflow tasks. During workflow execution, Sizey simultaneously trains multiple machine learning models and then dynamically selects the best model for each workflow task. To evaluate the quality of the model, we introduce a novel resource allocation quality (RAQ) score based on memory prediction accuracy and efficiency. Sizey's prediction models are retrained and re-evaluated online during workflow execution, continuously incorporating metrics from comnleted tasks. Our evaluation with a prototype implementation of Sizey uses metrics from six real-world scientific workflows from the popular nf-core framework and shows a median reduction in memory waste over time of 24.68 % compared to the respective bestnerforming state-of-the-art baseline.

Problem

• Memory-efficient execution of workflows
• Online memory prediction for tasks
• Reduction in memory waste

Innovation

• Online memory prediction method
• Dynamic machine learning model selection
• Resource allocation quality score