Agentic Orchestration of HPC Applications in Cloud

2026-07-02

AI Summary

This work proposes the first large language model (LLM)-driven agent framework for autonomous, end-to-end management of high-performance computing (HPC) applications in cloud environments. Addressing the heavy reliance on manual intervention and the lack of intelligent decision-making in traditional HPC cloud deployment, the framework enables automated multi-platform container construction, Kubernetes-based orchestration, cross-instance performance optimization, and adaptive elastic scaling policy generation. By integrating LLM-powered agents into HPC cloud workflow orchestration, this study establishes a novel paradigm of automation and self-adaptation. Experimental evaluation across four representative HPC applications demonstrates that the system achieves expert-level linear scalability, substantially reduces job completion time, and yields actionable best practices for collaborative agent design in HPC contexts.

Abstract

Large Language Models (LLMs) are serving as a catalyst of change for research practices, touching the daily lives of staff scientists, software engineers, and system administrators. The developments promise new degrees of autonomy, where categories of human work and decision making are replaced by autonomous, goal-oriented systems. This transition necessitates novel architectural paradigms and solid understanding of the strengths and limitations of LLMs. In this work, we design agents to intelligently deliver the entire life-cycle of an HPC application experimental run in cloud -- creation and build of a container, deployment in Kubernetes, optimization, and orchestration of a scaling study. We pursue this task for four well-known HPC applications to build multi-platform images and optimize across 21 instance types in Kubernetes. We demonstrate successful linear scaling with patterns approved by human experts, designs that improve work time to completion, and review suggested best practices for agentic design and collaboration.

Problem

• Agentic Orchestration
• HPC Applications
• Cloud Computing
• LLM-based Automation
• Kubernetes

Innovation

• Agentic Orchestration
• Large Language Models
• HPC Applications
• Cloud Computing
• Kubernetes