Big Data Workload Profiling for Energy-Aware Cloud Resource Management

2026-01-17

AI Summary

This work addresses the challenges of high energy consumption and inefficient resource scheduling in cloud data centers under big data workloads. The authors propose an adaptive resource scheduling mechanism based on workload profiling, which integrates historical logs and real-time telemetry data to model CPU, memory, and storage I/O characteristics. This model predicts the energy and performance implications of virtual machine placement decisions and dynamically triggers resource consolidation. Evaluated on a multi-node cloud platform using representative workloads—including Hadoop MapReduce, Spark MLlib, and ETL pipelines—the approach achieves consistent energy savings of 15%–20% over baseline schedulers while incurring negligible performance degradation, all while adhering to service-level agreements. The method thus effectively balances energy efficiency and sustainability without compromising operational performance.

Technology Category

• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems
• Machine Learning: Efficient ML / Green AI
• Planning, Routing, and Scheduling: Scheduling

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Virtualization and resource management in Web systems and infrastructures
• Economics, Online Markets and Human Computation: Sustainability of Web economics
• Responsible Web: Sustainability and climate impact of web technologies

Abstract

Cloud data centers face increasing pressure to reduce operational energy consumption as big data workloads continue to grow in scale and complexity. This paper presents a workload aware and energy efficient scheduling framework that profiles CPU utilization, memory demand, and storage IO behavior to guide virtual machine placement decisions. By combining historical execution logs with real time telemetry, the proposed system predicts the energy and performance impact of candidate placements and enables adaptive consolidation while preserving service level agreement compliance. The framework is evaluated using representative Hadoop MapReduce, Spark MLlib, and ETL workloads deployed on a multi node cloud testbed. Experimental results demonstrate consistent energy savings of 15 to 20 percent compared to a baseline scheduler, with negligible performance degradation. These findings highlight workload profiling as a practical and scalable strategy for improving the sustainability of cloud based big data processing environments.

Problem

• Big Data
• Energy Consumption
• Cloud Resource Management
• Workload Profiling
• Virtual Machine Placement

Innovation

• workload profiling
• energy-aware scheduling
• virtual machine placement
• big data workloads
• cloud resource management