Descriptive Dispatch of Computational Work

2026-08-11

AI Summary

This work addresses the challenges of scheduling scientific workflows in multi-cluster environments, where dynamic task transformation, heterogeneous resource matching, and reliable job dispatch are critical. The authors propose an intelligent agent framework powered by large language models that leverages multidimensional prompt engineering to parse natural-language instructions and automatically generate job specifications. Integrated with a metadata-driven dynamic transformation mechanism, the framework enables end-to-end automation of cross-cluster scheduling. Experimental results demonstrate a 97.9% scheduling success rate across 432 runs; incorporating descriptive metadata increased the successful execution rate of 220 jobs from 48% to 87%, while five representative applications achieved up to a 3.3× performance speedup, effectively resolving architectural mismatches and enabling efficient, reliable task distribution.

Abstract

Agents powered by AI/ML are becoming ingrained in orchestration. Dispatch of work is the task of receiving a request, transforming it for a workload manager, and successfully submitting it. Running scientific workflows across multi-cluster environments introduces substantial challenges of dynamic job transformation, dispatch, and submission to heterogeneous clusters. These tasks are well-suited to agents, which can receive textual instructions for work, prepare job specifications, and dispatch. In this work, we assess the reliability of a dispatch agent across 432 runs, testing all possible combinations of five feature dimensions across four prompt styles. The agent is highly reliable (97.9% success). We test a full orchestration to submit, queue, match, score, select, transform, and dispatch in a multi-cluster experiment. We find that descriptive metadata increases successful execution from 48% to 87% of 220 submitted jobs, eliminating architecture mismatch, and improving performance for five of ten measurable applications by up to 3.3x.

Problem

• multi-cluster
• scientific workflows
• job dispatch
• heterogeneous clusters
• architecture mismatch

Innovation

• Descriptive Dispatch
• AI Agent
• Multi-cluster Orchestration
• Job Transformation
• Metadata-driven Scheduling