Does Functional Package Management Enable Reproducible Builds at Scale? Yes

2025-01-27

AI Summary

Bitwise reproducibility in large-scale open-source software distribution remains challenging due to non-deterministic build artifacts—especially timestamp embedding—and lacks comprehensive empirical assessment. Method: This work conducts the first seven-year (2017–2023), large-scale empirical study of bitwise reproducibility across 709,000 packages in nixpkgs—the largest functional package repository—using Nix. We design an automated rebuild pipeline integrating diffoscope-based recursive binary diffing and root-cause classification. Contribution/Results: We achieve 69%–91% bitwise reproducibility (with a strong upward trend) and >99% overall rebuild success. Timestamp embedding accounts for 15% of irreproducible cases. We release the first open dataset containing full differential analysis, build logs, and reproducibility status—enabling the largest benchmark and most rigorous methodology to date for reproducible builds research.

Technology Category

• Data Mining & Knowledge Management: Representing, Reasoning, and Using Provenance, Trust
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Humans and AI: Other Foundations of Human Computation & AI

Application Category

• Security and Privacy: Data transparency and provenance
• Web Mining and Content Analysis: Web data provenance, reliability, and authenticity
• Responsible Web: Sustainability and climate impact of web technologies

Abstract

Reproducible Builds (R-B) guarantee that rebuilding a software package from source leads to bitwise identical artifacts. R-B is a promising approach to increase the integrity of the software supply chain, when installing open source software built by third parties. Unfortunately, despite success stories like high build reproducibility levels in Debian packages, uncertainty remains among field experts on the scalability of R-B to very large package repositories. In this work, we perform the first large-scale study of bitwise reproducibility, in the context of the Nix functional package manager, rebuilding 709 816 packages from historical snapshots of the nixpkgs repository, the largest cross-ecosystem open source software distribution, sampled in the period 2017-2023. We obtain very high bitwise reproducibility rates, between 69 and 91% with an upward trend, and even higher rebuildability rates, over 99%. We investigate unreproducibility causes, showing that about 15% of failures are due to embedded build dates. We release a novel dataset with all build statuses, logs, as well as full ''diffoscopes'': recursive diffs of where unreproducible build artifacts differ.

Problem

• Software Package Repositories
• Reproducible Building
• Temporal Dependencies

Innovation

• Reproducible Building (R-B)
• Nix Package Manager
• Deterministic Build Process