Benchmarking Recommendation, Classification, and Tracing Based on Hugging Face Knowledge Graph

2025-05-23

AI Summary

Existing ML platforms (e.g., Hugging Face) lack structured knowledge support, hindering advanced information retrieval (IR) tasks such as model evolution tracking and cross-resource recommendation. Method: We introduce HuggingKG—the first large-scale ML resource knowledge graph (KG) for the Hugging Face ecosystem—comprising 2.6M nodes and 6.2M edges—and release HuggingBench, a multi-task IR benchmark covering resource recommendation, fine-grained classification, and evolutionary path tracing. Contribution/Results: This work pioneers the formalization of open ML communities as domain-specific KGs; defines novel IR tasks—including model and dataset evolution tracing; and open-sources the first integrated KG-and-benchmark infrastructure for ML resource management. Experiments demonstrate that HuggingKG substantially enhances semantic representation and cross-task transferability, while HuggingBench exposes critical performance bottlenecks of current IR models in cross-type evolutionary tracing. All resources are publicly available.

Technology Category

• Machine Learning: Graph-based Machine Learning
• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion
• Knowledge Representation and Reasoning: Knowledge Representation Languages

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

The rapid growth of open source machine learning (ML) resources, such as models and datasets, has accelerated IR research. However, existing platforms like Hugging Face do not explicitly utilize structured representations, limiting advanced queries and analyses such as tracing model evolution and recommending relevant datasets. To fill the gap, we construct HuggingKG, the first large-scale knowledge graph built from the Hugging Face community for ML resource management. With 2.6 million nodes and 6.2 million edges, HuggingKG captures domain-specific relations and rich textual attributes. It enables us to further present HuggingBench, a multi-task benchmark with three novel test collections for IR tasks including resource recommendation, classification, and tracing. Our experiments reveal unique characteristics of HuggingKG and the derived tasks. Both resources are publicly available, expected to advance research in open source resource sharing and management.

Problem

• Lack of structured representations for ML resources on platforms like Hugging Face
• Need for advanced queries and analyses such as model evolution tracing
• Absence of a large-scale knowledge graph for ML resource management

Innovation

• Constructs HuggingKG from Hugging Face data
• Enables multi-task benchmark HuggingBench
• Supports recommendation, classification, tracing tasks