Enabling Seamless Transitions from Experimental to Production HPC for Interactive Workflows

2025-06-02

AI Summary

Experimental HPC environments struggle to support seamless migration of interactive scientific workflows into production systems. Method: This paper proposes a tripartite transition framework integrating dataflow-driven execution, zero-trust security interfaces, and QoS-aware adaptive scheduling. It innovatively unifies dynamic dataflow architecture, zero-trust–based secure service interfaces, and elastic, timeliness-sensitive resource scheduling to enable the paradigm shift from batch-oriented HPC to near-real-time interactive ecosystems. Contribution/Results: Evaluated on the Oak Ridge Leadership Computing Facility (OLCF) platform, the framework achieves a 40% reduction in end-to-end latency, sub-second interactive response times, and enables multi-disciplinary real-time closed-loop experiments. It has been deployed in production on the Summit and Frontier exascale supercomputers, providing a reusable, structured migration pathway for cross-facility collaboration, dynamic experimental control, and near-real-time analysis.

Technology Category

• Machine Learning: Hardware-aware ML
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Security and Privacy: Data transparency and provenance
• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

The evolving landscape of scientific computing requires seamless transitions from experimental to production HPC environments for interactive workflows. This paper presents a structured transition pathway developed at OLCF that bridges the gap between development testbeds and production systems. We address both technological and policy challenges, introducing frameworks for data streaming architectures, secure service interfaces, and adaptive resource scheduling for time-sensitive workloads and improved HPC interactivity. Our approach transforms traditional batch-oriented HPC into a more dynamic ecosystem capable of supporting modern scientific workflows that require near real-time data analysis, experimental steering, and cross-facility integration.

Problem

• Bridging experimental and production HPC for interactive workflows
• Addressing technological and policy challenges in HPC transitions
• Enhancing HPC interactivity for real-time data analysis

Innovation

• Structured transition pathway for HPC environments
• Frameworks for data streaming and secure interfaces
• Adaptive resource scheduling for real-time workloads