🤖 AI Summary
To address the insufficient real-time adaptability of task scheduling under node affinity constraints in HPC clusters, this paper proposes a dynamic scheduling method based on continual transfer learning. Unlike traditional schedulers (e.g., Kubernetes) that require frequent offline retraining, our approach enables lightweight, online model evolution using incoming task streams—eliminating the need for periodic retraining and significantly reducing operational overhead. Empirical evaluation on the Google Cluster Trace demonstrates that the method maintains over 99% scheduling prediction accuracy while reducing average scheduling latency for affinity-constrained tasks and lowering overall system computational cost. The core contribution is the integration of a continual transfer learning mechanism into the scheduling decision loop, establishing the first affinity-aware, low-overhead, and high-temporal-fidelity adaptive scheduling framework. This advances scalability and dynamic responsiveness in large-scale HPC systems.
📝 Abstract
This study presents a machine learning-assisted approach to optimize task scheduling in cluster systems, focusing on node-affinity constraints. Traditional schedulers like Kubernetes struggle with real-time adaptability, whereas the proposed continuous transfer learning model evolves dynamically during operations, minimizing retraining needs. Evaluated on Google Cluster Data, the model achieves over 99% accuracy, reducing computational overhead and improving scheduling latency for constrained tasks. This scalable solution enables real-time optimization, advancing machine learning integration in cluster management and paving the way for future adaptive scheduling strategies.