🤖 AI Summary
This work addresses the challenge of real-time human oversight in high-performance computing (HPC) environments, where manual intervention is often impractical and can lead to resource waste and delayed decisions. To overcome this, the paper introduces the first workflow-oriented asynchronous human-in-the-loop framework, which enables non-blocking collaboration by pausing at designated checkpoints to solicit human input while allowing underlying tasks to continue execution across hybrid infrastructures—spanning HPC systems, on-premises clusters, and cloud platforms. The framework integrates with the SLURM scheduler, supports both containerized and native workloads, and leverages a cross-platform workflow engine combined with checkpointing technology. Experiments on systems such as MareNostrum 5 demonstrate significant improvements in portability, computational efficiency, and supervisory control, particularly benefiting high-stakes scenarios requiring flexible human–machine coordination.
📝 Abstract
Human involvement is critical in training and deploying AI systems in high-stakes defence and security contexts. However, real-time interaction is impractical in HPC environments due to compute intensity and resource constraints. We present a workflow framework that enables asynchronous human-AI collaboration across hybrid infrastructures, including HPC clusters, local machines, and cloud platforms. Workflows can pause at defined checkpoints for human input without halting underlying compute jobs, preventing idle resources and enabling non-blocking supervision. The framework supports interaction with SLURM-based scheduling, containerized and native tasks, and is customized for scenarios requiring human judgment and adaptability. We demonstrate its application in model training on systems like MareNostrum 5, highlighting benefits in portability, efficiency, and oversight in operational AI workflows.