Carbon- and Precedence-Aware Scheduling for Data Processing Clusters

2025-02-13

AI Summary

Carbon-aware scheduling in big-data processing is hindered by the misalignment between task dependencies (modeled as DAGs) and dynamic, low-carbon energy supply. Method: This paper proposes the first carbon-aware scheduling framework that explicitly models task priority constraints. It introduces a configurable carbon–timeliness trade-off mechanism, theoretically characterizing its Pareto frontier. We implement PCAPS—a production-ready scheduler—and CAP—a lightweight adapter—integrating time-varying carbon intensity forecasting, DAG-aware dependency modeling, and ML-driven policy interfaces, with native support for Kubernetes and Spark. Results: Evaluated on a 100-node cluster, PCAPS reduces carbon footprint by up to 32.9%, achieves zero increase in job completion time, and maintains throughput efficiency without statistically significant degradation.

Technology Category

• Planning, Routing, and Scheduling: Scheduling under Uncertainty
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Machine Learning: Hardware-aware ML

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Sustainability and carbon-aware systems for Web, mobile, and WoT
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

As large-scale data processing workloads continue to grow, their carbon footprint raises concerns. Prior research on carbon-aware schedulers has focused on shifting computation to align with availability of low-carbon energy, but these approaches assume that each task can be executed independently. In contrast, data processing jobs have precedence constraints (i.e., outputs of one task are inputs for another) that complicate decisions, since delaying an upstream ``bottleneck'' task to a low-carbon period will also block downstream tasks, impacting the entire job's completion time. In this paper, we show that carbon-aware scheduling for data processing benefits from knowledge of both time-varying carbon and precedence constraints. Our main contribution is $ exttt{PCAPS}$, a carbon-aware scheduler that interfaces with modern ML scheduling policies to explicitly consider the precedence-driven importance of each task in addition to carbon. To illustrate the gains due to fine-grained task information, we also study $ exttt{CAP}$, a wrapper for any carbon-agnostic scheduler that adapts the key provisioning ideas of $ exttt{PCAPS}$. Our schedulers enable a configurable priority between carbon reduction and job completion time, and we give analytical results characterizing the trade-off between the two. Furthermore, our Spark prototype on a 100-node Kubernetes cluster shows that a moderate configuration of $ exttt{PCAPS}$ reduces carbon footprint by up to 32.9% without significantly impacting the cluster's total efficiency.

Problem

• carbon-aware scheduling
• precedence constraints
• data processing clusters

Innovation

• Carbon-aware scheduling with precedence
• PCAPS integrates ML scheduling policies
• CAP adapts PCAPS for carbon-agnostic systems