How Accurately Can the Energy Use of Spark Applications Be Estimated Based on Resource Utilisation?

2026-08-10

AI Summary

This study addresses the challenge of inaccurate energy consumption estimation for distributed batch-processing applications like Apache Spark in cloud environments, where node-level hardware energy counters are typically inaccessible. Focusing on Apache Spark deployed on Kubernetes, the work presents the first systematic comparison between resource-utilization-based energy models and ground-truth measurements from Intel RAPL across both AWS bare-metal instances and on-premises clusters. It investigates the impact of CPU and memory utilization signals on estimation accuracy and introduces external monitoring to enhance model fidelity. Experimental results demonstrate that incorporating external monitoring significantly mitigates energy underestimation—reducing the error from −29.58% to −24.41% on AWS and from −24.00% to −16.22% in the local cluster—thereby validating its effectiveness in improving energy estimation accuracy.

Abstract

Distributed batch data processing applications are widely executed on cloud-based resources where restricted user access to node-level hardware energy counters hinders transparent sustainability accounting. Energy and carbon attribution methodologies therefore depend on power models and available resource utilisation traces, yet the accuracy of these estimates has to be validated while direct counters are available. In this work, we use Apache Spark running on Kubernetes as a case-study dataflow runtime and cluster resource manager to compare model-based energy estimates to Intel RAPL package and DRAM energy on an AWS bare-metal cloud and an on-premises cluster, comparing different CPU usage signals and memory coefficients. We show that external monitoring improves signed package-energy error relative to Spark task traces, reducing underestimation from -29.58% to -24.41% on AWS and from -24.00% to -16.22% on-premises.

Problem

• energy estimation
• resource utilisation
• distributed data processing
• cloud computing
• power modeling

Innovation

• energy estimation
• resource utilisation
• Apache Spark
• RAPL
• cloud sustainability