Scalable Runtime Architecture for Data-driven, Hybrid HPC and ML Workflow Applications

2025-03-17

AI Summary

To address the challenges of cross-platform orchestration and fragmented resource scheduling in hybrid HPC–ML workflows, this paper proposes a service-oriented, scalable runtime architecture. Building upon the RADICAL-Pilot framework, we introduce the first service-oriented execution model enabling dynamic, multi-granularity, low-overhead coordination of heterogeneous HPC and ML tasks. Our approach unifies resource abstraction across platforms, implements distributed task scheduling, and jointly orchestrates AI and HPC workloads—thereby enabling seamless coupling and coordinated scheduling between on-premises exascale supercomputers and cloud environments. Experimental evaluation on an exascale prototype system demonstrates concurrent deployment of multiple ML models with runtime overhead under 2%. The architecture successfully supports three representative data-driven scientific applications, effectively overcoming the traditional siloing of HPC and ML workflows.

Technology Category

• Machine Learning: Hardware-aware ML
• Humans and AI: Planning and Decision Support for Human-Machine Teams
• Planning, Routing, and Scheduling: Mixed Discrete/Continuous Planning

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work
• Security and Privacy: Security and privacy of machine learning and AI applications

Abstract

Hybrid workflows combining traditional HPC and novel ML methodologies are transforming scientific computing. This paper presents the architecture and implementation of a scalable runtime system that extends RADICAL-Pilot with service-based execution to support AI-out-HPC workflows. Our runtime system enables distributed ML capabilities, efficient resource management, and seamless HPC/ML coupling across local and remote platforms. Preliminary experimental results show that our approach manages concurrent execution of ML models across local and remote HPC/cloud resources with minimal architectural overheads. This lays the foundation for prototyping three representative data-driven workflow applications and executing them at scale on leadership-class HPC platforms.

Problem

• Develop scalable runtime for hybrid HPC and ML workflows
• Enable distributed ML and efficient resource management
• Support seamless HPC/ML coupling across local and remote platforms

Innovation

• Extends RADICAL-Pilot with service-based execution
• Enables distributed ML and efficient resource management
• Supports seamless HPC/ML coupling across platforms