Understanding the Effect of Knowledge Graph Extraction Error on Downstream Graph Analyses: A Case Study on Affiliation Graphs

2025-06-14

AI Summary

This work systematically investigates how knowledge graph (KG) extraction errors affect downstream macroscopic graph analytics—specifically community detection and connectivity analysis—in person-institution affiliation graphs. We propose a dual-level evaluation framework assessing both micro-level edge accuracy and macro-level graph metrics (e.g., modularity, number of connected components), integrating LLM-based KG extraction, human-driven error attribution, controlled error injection, and large-scale simulation experiments. Our key finding is the first empirical identification of a nonlinear relationship between extraction accuracy and graph metric deviation: a critical accuracy threshold exists, above which metric bias vanishes asymptotically, and below which stable directional biases—systematic overestimation or underestimation—emerge. Crucially, standard error models fail to reproduce this empirically observed bias pattern. These results provide both theoretical foundations and practical guidelines for KG quality assessment and robust graph analytics.

Technology Category

• Data Mining & Knowledge Management: Graph Mining, Social Network Analysis & Community
• Machine Learning: Graph-based Machine Learning
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Knowledge graphs (KGs) are useful for analyzing social structures, community dynamics, institutional memberships, and other complex relationships across domains from sociology to public health. While recent advances in large language models (LLMs) have improved the scalability and accessibility of automated KG extraction from large text corpora, the impacts of extraction errors on downstream analyses are poorly understood, especially for applied scientists who depend on accurate KGs for real-world insights. To address this gap, we conducted the first evaluation of KG extraction performance at two levels: (1) micro-level edge accuracy, which is consistent with standard NLP evaluations, and manual identification of common error sources; (2) macro-level graph metrics that assess structural properties such as community detection and connectivity, which are relevant to real-world applications. Focusing on affiliation graphs of person membership in organizations extracted from social register books, our study identifies a range of extraction performance where biases across most downstream graph analysis metrics are near zero. However, as extraction performance declines, we find that many metrics exhibit increasingly pronounced biases, with each metric tending toward a consistent direction of either over- or under-estimation. Through simulations, we further show that error models commonly used in the literature do not capture these bias patterns, indicating the need for more realistic error models for KG extraction. Our findings provide actionable insights for practitioners and underscores the importance of advancing extraction methods and error modeling to ensure reliable and meaningful downstream analyses.

Problem

• Evaluates impact of KG extraction errors on downstream analyses
• Assesses micro-level edge accuracy and macro-level graph metrics
• Identifies biases in graph metrics due to extraction errors

Innovation

• Evaluates KG extraction errors at micro and macro levels
• Identifies bias patterns in downstream graph metrics
• Proposes need for realistic KG extraction error models