Benchmarking Different Application Types across Heterogeneous Cloud Compute Services

2025-01-10

AI Summary

This study addresses the lack of cross-platform, multidimensional performance evaluation in enterprise cloud service selection by introducing the first heterogeneous cloud computing benchmark framework focused on energy efficiency and sustainability. Methodologically, we deploy three representative workloads—industrial-grade DNN inference, assistive-technology ML inference, and video transcoding—across real-world cloud environments (AWS, Azure, GCP), measuring cost, latency, throughput, power consumption, and carbon emissions. Our key contribution is the first systematic, three-dimensional evaluation spanning application type, hardware architecture (ARM/x86/GPU), and cloud provider, complemented by an open-source, standardized benchmark dataset. Results uncover architecture-specific performance–efficiency trade-offs and provide empirical foundations for green cloud resource scheduling and low-carbon compute decision-making.

Technology Category

• Machine Learning: Efficient ML / Green AI
• Humans and AI: Other Foundations of Human Computation & AI
• Search and Optimization: Evaluation and Analysis

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Sustainability and carbon-aware systems for Web, mobile, and WoT
• Responsible Web: Sustainability and climate impact of web technologies
• Economics, Online Markets and Human Computation: Sustainability of Web economics

Abstract

Infrastructure as a Service (IaaS) clouds have become the predominant underlying infrastructure for the operation of modern and smart technology. IaaS clouds have proven to be useful for multiple reasons such as reduced costs, increased speed and efficiency, and better reliability and scalability. Compute services offered by such clouds are heterogeneous -- they offer a set of architecturally diverse machines that fit efficiently executing different workloads. However, there has been little study to shed light on the performance of popular application types on these heterogeneous compute servers across different clouds. Such a study can help organizations to optimally (in terms of cost, latency, throughput, consumed energy, carbon footprint, etc.) employ cloud compute services. At HPCC lab, we have focused on such benchmarks in different research projects and, in this report, we curate those benchmarks in a single document to help other researchers in the community using them. Specifically, we introduce our benchmarks datasets for three application types in three different domains, namely: Deep Neural Networks (DNN) Inference for industrial applications, Machine Learning (ML) Inference for assistive technology applications, and video transcoding for multimedia use cases.

Problem

• Cloud Computing Services
• Performance Evaluation
• Deep Neural Network Inference

Innovation

• Cloud Computing Performance
• Deep Neural Network Inference
• Machine Learning Inference