Beyond Compliance: A Large Scale Study on the Completeness and Consistency of the GitHub SBOMs

2026-07-05

AI Summary

This study presents the first systematic evaluation of the reliability and cross-ecosystem consistency of GitHub’s automatically generated Software Bill of Materials (SBOM) for vulnerability and license analysis. Through large-scale static analysis of 10,000 repositories, the authors compare GitHub’s SBOM against those produced by Syft, Trivy, and Microsoft’s SBOM tool across ten programming language ecosystems, assessing component identification, version extraction, and license detection capabilities. The findings indicate that, although GitHub’s SBOM does not yet fully comply with NTIA requirements, its core metadata remains stable and consistently outperforms Syft and Trivy in version and license coverage across most ecosystems, approaching the performance of Microsoft’s tool. The results further reveal that dependency management mechanisms are a critical determinant of SBOM quality.

Abstract

Modern software development relies heavily on open-source components. Reusing components accelerates innovation but increases exposure to supply-chain attacks exploiting known vulnerabilities. Software Bills of Materials (SBOMs) improve software supply chain transparency by enumerating components, their versions, and their provenance. GitHub, the largest open-source development hosting platform, now automatically generates SBOMs for repositories, providing valuable metadata for risk assessment. Yet, it is unclear whether GitHub SBOMs can serve as a reliable source for vulnerability and license analysis, and how incomplete or inconsistent metadata may affect different programming ecosystems. To address this, we conduct a large-scale analysis of 10,000 GitHub repositories across ten programming language ecosystems, evaluating GitHub SBOMs against three other popular SBOM generators: Syft, Trivy, and the Microsoft SBOM Tool. Our study finds a lack of NTIA compliance in GitHub SBOMs, though core metadata is consistently present. We also find that component version and license information availability is highly dependent on the programming ecosystem. Compared with the other three tools, GitHub yields results similar to the Microsoft SBOM Tool and often outperforms Syft and Trivy in providing version and license information. Finally, we discuss potential shortcomings of the GitHub SBOM Tool, directly related to how each ecosystem manages its dependencies.

Problem

• SBOM
• software supply chain
• metadata completeness
• metadata consistency
• open-source dependencies

Innovation

• Software Bill of Materials
• GitHub SBOM
• supply chain security
• dependency metadata
• ecosystem comparison