KGCQual: An Interpretable Framework for Evaluating the Knowledge Graph Construction Quality from Text

2026-07-11

AI Summary

This work addresses the challenge of evaluating automatically constructed knowledge graphs, which often contain erroneous or missing triples, by proposing KGCQual—an interpretable, intrinsic evaluation framework that jointly assesses structural and semantic fidelity. KGCQual measures how closely a knowledge graph approximates an ideal textual representation along two granular dimensions: entity-level qualities (completeness, parsing quality, and connectivity) and relation-level properties (predicate preservation, multiplicity, and negation handling). The approach integrates lexical similarity, dependency-based syntactic alignment, and lightweight negation detection, making it model-agnostic and scalable. Experiments on WebNLG, TinyButMighty, and BenchIE demonstrate that KGCQual effectively identifies omissions, redundancies, and structural biases, with its scores showing significant correlation to downstream link prediction performance.

Abstract

Knowledge Graphs (KGs) are increasingly constructed through automated extraction pipelines; however, such systems often introduce spurious or incomplete triples, which degrade downstream performance. Existing evaluation practices rely heavily on task-specific metrics or small-scale manual verification, offering limited insight into the structural and semantic fidelity of extracted graphs. We propose a novel, interpretable metric for intrinsic KG quality assessment that measures how closely an automatically extracted graph approximates an "ideal" graph capturing the key noun phrases, predicate relations, and basic linguistic phenomena such as negation expressed in the source text. Our framework integrates two complementary components: (1) an entity-level assessment that evaluates completeness, resolution quality, and connectivity, and (2) a relation-level assessment that judges predicate preservation and multiplicity using lexical similarity, dependency-parse alignment, and light-weight negation handling to ensure semantic faithfulness. We evaluate our metric across multiple state-of-the-art triple extraction systems and datasets, including WebNLG, TinyButMighty, and BenchIE, demonstrating that it reliably identifies omissions, redundancy, and structural deviations that existing metrics overlook. Our work offers a scalable, model-agnostic, and interpretable framework for comparing automated KG construction methods and provides a foundation for standardised evaluation. We further validate the metric through an ablation study isolating noun and verb components, and a downstream evaluation showing that KGCQual scores correlate significantly with link prediction performance on the same extracted KGs. The code repository is available at https://github.com/kracr/kg-quality-metric.

Problem

• Knowledge Graph Construction
• Quality Evaluation
• Triple Extraction
• Semantic Fidelity
• Structural Integrity

Innovation

• Knowledge Graph Construction
• Interpretable Evaluation
• Intrinsic Quality Metric
• Relation Faithfulness
• Entity Resolution