The Cost of Downgrading Build Systems: A Case Study of Kubernetes

2025-10-22

AI Summary

This study investigates the performance trade-offs of downgrading the build system from Bazel to `go build` in large-scale projects (e.g., Kubernetes), a common but poorly understood engineering decision. Method: We conduct full and incremental build reproductions, perform fine-grained resource monitoring (CPU, memory), and empirically compare across multiple projects—including Kubernetes and Istio—to quantify changes in build time, resource consumption, and CI infrastructure cost. Results: While Bazel delivers faster builds, it incurs substantially higher memory and CPU overhead. Downgrading increases average single-build latency by 2.3×; counterintuitively, CI cluster resource costs rise by 76%, a trend consistently observed across projects. This work is the first to systematically expose the hidden costs of build-system downgrades. It provides a reproducible, scalable, quantitative framework for build-tool selection—filling a critical gap in empirical evidence for tooling evolution in large-scale software engineering practice.

Technology Category

• Machine Learning: Scalability of ML Systems
• Constraint Satisfaction and Optimization: Solvers and Tools
• Humans and AI: Other Foundations of Human Computation & AI

Application Category

• Security and Privacy: Large-scale security measurements
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization

Abstract

Since developers invoke the build system frequently, its performance can impact productivity. Modern artifact-based build tools accelerate builds, yet prior work shows that teams may abandon them for alternatives that are easier to maintain. While prior work shows why downgrades are performed, the implications of downgrades remain largely unexplored. In this paper, we describe a case study of the Kubernetes project, focusing on its downgrade from an artifact-based build tool (Bazel) to a language-specific solution (Go Build). We reproduce and analyze the full and incremental builds of change sets during the downgrade period. On the one hand, we find that Bazel builds are faster than Go Build, completing full builds in 23.06-38.66 up to 75.19 impose a larger memory footprint than Go Build of 81.42-351.07 respectively. Bazel builds also impose a greater CPU load at parallelism settings above eight for full builds and above one for incremental builds. We estimate that downgrading from Bazel can increase CI resource costs by up to 76 explore whether our observations generalize by replicating our Kubernetes study on four other projects that also downgraded from Bazel to older build tools. We observe that while build time penalties decrease, Bazel consistently consumes more memory. We conclude that abandoning artifact-based build tools, despite perceived maintainability benefits, tends to incur considerable performance costs for large projects. Our observations may help stakeholders to balance trade-offs in build tool adoption

Problem

• Analyzing performance costs when downgrading from Bazel to Go Build
• Comparing build speed, memory usage, and CPU load between systems
• Investigating trade-offs between maintainability and build performance

Innovation

• Analyzed downgrade from Bazel to Go Build
• Reproduced full and incremental build performance
• Compared resource costs across multiple projects