Bridging the Gap Between Binary and Source Based Package Management in Spack

2025-09-09

AI Summary

In HPC environments, binary package managers suffer from inflexible configuration due to strict ABI constraints, while source-based builds incur inefficiency from redundant compilation (e.g., when switching MPI implementations). Although Spack supports hybrid source/binary deployment, its lack of a formal binary compatibility model prevents safe reuse of binaries built across heterogeneous environments. This work introduces *splicing*, a novel Spack extension that establishes the first fine-grained ABI compatibility model. It integrates metadata annotation, enhanced dependency resolution, and runtime compatibility checking to enable secure cross-environment binary reuse. The approach seamlessly unifies the configurability of source builds with the efficiency of binary installation—reducing redundant compilations significantly, achieving near-binary installation speed, and incurring negligible overhead. As a result, splicing enhances both flexibility and scalability of HPC software distribution.

Technology Category

• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Virtualization and resource management in Web systems and infrastructures
• Security and Privacy: Data transparency and provenance
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Binary package managers install software quickly but they limit configurability due to rigid ABI requirements that ensure compatibility between binaries. Source package managers provide flexibility in building software, but compilation can be slow. For example, installing an HPC code with a new MPI implementation may result in a full rebuild. Spack, a widely deployed, HPC-focused package manager, can use source and pre-compiled binaries, but lacks a binary compatibility model, so it cannot mix binaries not built together. We present splicing, an extension to Spack that models binary compatibility between packages and allows seamless mixing of source and binary distributions. Splicing augments Spack's packaging language and dependency resolution engine to reuse compatible binaries but maintains the flexibility of source builds. It incurs minimal installation-time overhead and allows rapid installation from binaries, even for ABI-sensitive dependencies like MPI that would otherwise require many rebuilds.

Problem

• Spack lacks binary compatibility model for mixing packages
• Binary and source package management have configurability trade-offs
• HPC installations require slow rebuilds for ABI-sensitive dependencies

Innovation

• Splicing enables binary compatibility modeling
• Seamlessly mixes source and binary distributions
• Minimal overhead with flexible dependency resolution