A Dataset of Software Bill of Materials for Evaluating SBOM Consumption Tools

2025-04-09

AI Summary

A lack of publicly available benchmark datasets for evaluating SBOM consumption tools hinders research and practical advancement in software supply chain security. To address this gap, we present the first high-quality, open-source dataset specifically designed for SBOM consumption tool evaluation. It comprises 46 Java projects, all rigorously compliant with the SPDX Lite specification and containing complete, validated entries for component names, versions, licenses, and other critical fields. SBOMs were generated at scale via Maven-integrated Syft and CycloneDX Maven Plugin, then enhanced through regex-based cleaning, dependency-resolution validation, and manual auditing. Crucially, we introduce a novel hybrid (automated + human) verification mechanism that not only improves SBOM accuracy but also uncovers inherent limitations in mainstream SBOM generation tools. The dataset is publicly archived on Zenodo (DOI: 10.5281/zenodo.14233415) to support functional evaluation, defect diagnosis, and tool refinement; future work will extend it to multi-language ecosystems.

Technology Category

• Constraint Satisfaction and Optimization: Solvers and Tools
• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion
• Application Domains: Software Engineering

Application Category

• Web Mining and Content Analysis: Web data provenance, reliability, and authenticity
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

A Software Bill of Materials (SBOM) is becoming an essential tool for effective software dependency management. An SBOM is a list of components used in software, including details such as component names, versions, and licenses. Using SBOMs, developers can quickly identify software components and assess whether their software depends on vulnerable libraries. Numerous tools support software dependency management through SBOMs, which can be broadly categorized into two types: tools that generate SBOMs and tools that utilize SBOMs. A substantial collection of accurate SBOMs is required to evaluate tools that utilize SBOMs. However, there is no publicly available dataset specifically designed for this purpose, and research on SBOM consumption tools remains limited. In this paper, we present a dataset of SBOMs to address this gap. The dataset we constructed comprises 46 SBOMs generated from real-world Java projects, with plans to expand it to include a broader range of projects across various programming languages. Accurate and well-structured SBOMs enable researchers to evaluate the functionality of SBOM consumption tools and identify potential issues. We collected 3,271 Java projects from GitHub and generated SBOMs for 798 of them using Maven with an open-source SBOM generation tool. These SBOMs were refined through both automatic and manual corrections to ensure accuracy, currently resulting in 46 SBOMs that comply with the SPDX Lite profile, which defines minimal requirements tailored to practical workflows in industries. This process also revealed issues with the SBOM generation tools themselves. The dataset is publicly available on Zenodo (DOI: 10.5281/zenodo.14233415).

Problem

• Lack of public SBOM dataset for evaluating consumption tools
• Need accurate SBOMs to assess tool functionality and issues
• Current SBOM generation tools have reliability challenges

Innovation

• Dataset of 46 SBOMs from Java projects
• Combined automatic and manual SBOM refinement
• Compliant with SPDX Lite profile standards