KG4RecEval: Does Knowledge Graph Really Matter for Recommender Systems?

2024-04-04

ACM Transactions on Information Systems

AI Summary

The necessity of knowledge graphs (KGs) in recommendation systems remains largely assumed rather than empirically validated, with prevailing consensus potentially overstating their utility—especially in cold-start scenarios. Method: We propose KG4RecEval, a systematic evaluation framework for KG-enhanced recommender systems (KG-RS), introducing Knowledge Graph Efficiency Ratio (KGER) as a novel metric to quantify the *actual performance gain* attributable to KG integration. We conduct rigorous KG degradation experiments—including KG removal, random perturbation, and sparsification—across four benchmark datasets and multiple state-of-the-art KG-RS models. Contribution/Results: Our analysis reveals that complete KG removal or random distortion induces no statistically significant drop in recommendation accuracy; moreover, KG contributions are substantially overestimated under cold-start conditions. This challenges the long-standing assumption that KG integrity is essential for performance. We establish the first reproducible, multi-scenario benchmark for quantifying KG value in recommendation, and identify key data- and model-level factors governing KGER, enabling more principled KG adoption.

Technology Category

• Data Mining & Knowledge Management: Recommender Systems
• Knowledge Representation and Reasoning: Knowledge Acquisition
• Machine Learning: Graph-based Machine Learning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Recommender systems (RSs) are designed to provide personalized recommendations to users. Recently, knowledge graphs (KGs) have been widely introduced in RSs to improve recommendation accuracy. In this study, however, we demonstrate that RSs do not necessarily perform worse even if the KG is downgraded to the user-item interaction graph only (or removed). We propose an evaluation framework KG4RecEval to systematically evaluate how much a KG contributes to the recommendation accuracy of a KG-based RS, using our defined metric KGER ( KG utilization efficiency in recommendation ). We consider the scenarios where knowledge in a KG gets completely removed, randomly distorted and decreased, and also where recommendations are for cold-start users. Our extensive experiments on four commonly used datasets and a number of state-of-the-art KG-based RSs reveal that: to remove, randomly distort or decrease knowledge does not necessarily decrease recommendation accuracy, even for cold-start users. These findings inspire us to rethink how to better utilize knowledge from existing KGs, whereby we discuss and provide insights into what characteristics of datasets and KG-based RSs may help improve KG utilization efficiency. The code and supplementary material of this paper are available at: https://github.com/HotBento/KG4RecEval .

Problem

• Knowledge Graph
• Recommendation System
• Information Incompleteness

Innovation

• KG4RecEval
• Recommendation Systems
• Knowledge Graphs Evaluation