cluster management

Addressing the escalating energy consumption and carbon emissions from large-model training and cloud service expansion, this paper investigates carbon-aware container scheduling. Through a systematic literature review, we present the first taxonomy of cloud-native schedulers—particularly Kubernetes—from an environmental sustainability perspective, integrating both hardware-centric and software-centric strategies. We propose the first comprehensive classification framework for cloud task scheduling explicitly targeting carbon reduction, explicitly characterizing each algorithm by its sustainability objective, optimization dimension, and technical approach. Our analysis identifies emerging trends—including dynamic carbon intensity awareness and multi-objective co-optimization—and highlights critical open challenges, such as real-time data-driven closed-loop control and cross-domain coordinated scheduling. This work provides theoretical foundations and practical guidelines for designing and standardizing low-carbon cloud systems.

In geographically distributed, multi-organizational scientific computing environments, centralized task schedulers (e.g., Kubernetes) struggle with cross-domain collaboration, infrastructure dynamism, and workflow-platform coupling. Method: We propose a decentralized control plane that leverages semantic naming to enable automatic, location-agnostic binding between computational requests and Kubernetes endpoints—eliminating the need for pre-configuration or location awareness. Our approach integrates lightweight service discovery with cross-cluster resource orchestration to support dynamic, adaptive scheduling across organizational boundaries. Contribution/Results: Experiments demonstrate that, without a global controller, our system significantly improves scheduling flexibility and cross-cluster workflow portability. It establishes a novel distributed scheduling paradigm for scientific computing—characterized by high adaptability, low platform coupling, and inherent support for heterogeneous, evolving infrastructures.

Traditional high-performance computing (HPC) and cloud computing have long remained siloed due to divergent origins, cultures, and technological trajectories, hindering their joint ability to address emerging heterogeneous scientific workloads demanding both agile service orchestration and ultra-low-latency, state-aware performance. Method: This paper introduces “Converged Computing,” a novel paradigm enabling co-deployment of the HPC workload manager Flux and user-space Kubernetes (Usernetes) on native supercomputing clusters. Leveraging Linux namespaces, cgroups, and a custom network plugin, it establishes an infrastructure-level convergence architecture. Contribution/Results: The approach unifies cloud-native automation and portability with HPC’s low-latency interconnects, high-bandwidth networking, and fine-grained resource scheduling. Experimental evaluation in hybrid environments demonstrates low-overhead execution of HPC applications and efficient cross-environment communication. An open-source, reproducible deployment framework is provided, offering a practical pathway for HPC centers to adopt cloud-native technologies.

This paper addresses the lack of clarity regarding the diversity and evolutionary trajectories of modern workload schedulers. We propose a cross-layer taxonomy comprising three categories: OS process scheduling, cluster job scheduling, and big-data scheduling. Through algorithmic feature analysis and historical comparative study, we systematically characterize the design rationales, optimization objectives, and technological evolution of these schedulers, uncovering shared design patterns across local and distributed environments. Our key contribution is the first unified classification framework, which identifies three fundamental differentiating dimensions: resource abstraction granularity, scheduling timing, and feedback mechanism. Based on this analysis, we distill general-purpose scheduling design principles targeting heterogeneity, scalability, and QoS guarantees. The study provides both theoretical foundations and practical guidance for scheduler selection, cross-layer coordination optimization, and next-generation scheduler architecture design.

In high-density Linux clusters, frequent CPU context switches cause significant performance degradation; even with optimal scheduler placement policies, excessive resource over-provisioning is commonly relied upon for mitigation—leading to substantial waste. This paper proposes a latency-aware group scheduling optimization: departing from traditional per-task fairness prioritization, it instead uses task completion latency as the primary scheduling objective. Leveraging dynamic cgroup workload characterization, it adaptively regulates runqueues and deeply modifies the Linux kernel scheduler to enable fine-grained, low-overhead group-level scheduling. Experimental evaluation demonstrates that, while strictly satisfying service-level agreement (SLA) constraints, the approach reduces cluster resource requirements by 28%, markedly improving resource utilization and overall system throughput.

SLURM logs in HPC scientific workflows lack explicit case identifiers, hindering direct application of process mining. Method: This paper proposes an automatic job-correlation method based on implicit job dependency modeling—parsing SLURM logs and jointly leveraging spatiotemporal job feature matching and graph-structured modeling to achieve end-to-end clustering of unannotated jobs. Contribution/Results: We introduce the first systematic preprocessing framework for process mining on HPC logs, integrating algorithms such as Heuristics Miner to support process discovery and bottleneck diagnosis. Evaluated on real-world HPC cluster logs, our approach significantly improves workflow traceability, accurately identifies I/O- and scheduler-related performance bottlenecks, and enables high-fidelity reconstruction of end-to-end process models.

This work addresses the high cognitive burden on users and excessive carbon emissions associated with scientific computing due to the complexity of the SLURM job scheduler interface and its lack of energy-aware scheduling mechanisms. To mitigate these issues, the authors propose a modular Perl-based toolkit featuring a simplified command-line interface and a text-based user interface (TUI) that supports job monitoring, cancellation, and automatic generation of specialized submission scripts. A key innovation is the introduction of an “eco-mode” that enables automatic energy-efficient scheduling through off-peak workload shifting. This approach significantly lowers the usability barrier, enhances job management efficiency, and effectively reduces the carbon footprint of research computing workflows.

Existing Kubernetes schedulers struggle to simultaneously optimize user-defined QoS objectives—such as energy efficiency, cost, and global performance—while lacking automated, declarative orchestration capabilities across heterogeneous cloud-fog-edge clusters. To address this, we propose the first QoS-aware federated orchestration system. Our approach employs a lightweight, Raft-replicated resource agent architecture tightly coupled with a centralized knowledge repository, enabling, for the first time, automatic translation of user-specified YAML-declared multi-dimensional QoS constraints (e.g., latency, energy consumption, cost) into microservice placement and dynamic migration policies. The system integrates Istio service mesh and federated cluster management to support policy-driven scheduling, QoS-compliant rescheduling, and zero-touch failover. Evaluated on a nine-cluster testbed, our system demonstrates both effectiveness and scalability in meeting diverse, cross-layer QoS requirements.

This work proposes the first large language model (LLM)-driven agent framework for autonomous, end-to-end management of high-performance computing (HPC) applications in cloud environments. Addressing the heavy reliance on manual intervention and the lack of intelligent decision-making in traditional HPC cloud deployment, the framework enables automated multi-platform container construction, Kubernetes-based orchestration, cross-instance performance optimization, and adaptive elastic scaling policy generation. By integrating LLM-powered agents into HPC cloud workflow orchestration, this study establishes a novel paradigm of automation and self-adaptation. Experimental evaluation across four representative HPC applications demonstrates that the system achieves expert-level linear scalability, substantially reduces job completion time, and yields actionable best practices for collaborative agent design in HPC contexts.

This work addresses the technical and behavioral challenges of transitioning from node-exclusive to resource-aware scheduling in production-grade heterogeneous HPC systems, a shift that risks disrupting established scientific workflows. To enable seamless, non-disruptive migration, the authors propose a collaborative operational framework integrating a time-bound compatibility layer, observability-driven feedback mechanisms, and targeted user guidance. Built upon Slurm’s TRES resource model, the approach combines runtime compatibility support, job queue monitoring, and user behavior analysis to preserve workflow continuity while substantially improving scheduling efficiency. Empirical results demonstrate dramatic reductions in median queue wait times—from 277 minutes to under 3 minutes for CPU jobs and from 81 minutes to 3.4 minutes for GPU jobs—alongside high long-term adoption rates among users who embraced the new submission paradigm.

Data-intensive applications in distributed cloud environments suffer performance degradation due to network congestion, asymmetric bandwidth, and cross-node data shuffling—factors inadequately captured by conventional host-resource–centric schedulers (e.g., CPU/memory-based). To address this, we propose the first supervised learning–driven, network-aware scheduling framework for multi-site clusters. Our approach integrates real-time Kubernetes node telemetry with FABRIC’s programmable network topology to train a Spark job execution time prediction model, enabling task-to-node matching and ranking. Its key innovation lies in the first application of supervised learning to real-time, geographically distributed, network-aware scheduling. Experimental evaluation demonstrates that our method improves optimal node selection accuracy by 34–54% over the default Kubernetes scheduler, significantly reducing data transfer latency and shortening job completion time.