Systematic Evaluation of Knowledge Graph Repair with Large Language Models

2025-07-30

AI Summary

Existing knowledge graph (KG) repair evaluation methods rely on dataset-specific benchmarks, suffering from poor reproducibility and limited generalizability. To address SHACL constraint violation repair, this paper introduces the first systematic evaluation framework: it designs a violation-induction mechanism to generate diverse, controllable, and reproducible constraint violation scenarios; and conducts end-to-end repair experiments leveraging large language models (LLMs) with multi-strategy prompt engineering. Our key innovation lies in the tight integration of SHACL semantics, graph-structural context, and prompt design. We demonstrate that prompts incorporating critical constraints and distilled contextual information significantly improve repair accuracy—achieving an average +23.6% gain over baselines. This work establishes a scalable, verifiable, and reproducible evaluation paradigm for KG repair, advancing both methodological rigor and practical applicability in constraint-aware KG maintenance.

Technology Category

• Knowledge Representation and Reasoning: Description Logics
• Planning, Routing, and Scheduling: Replanning and Plan Repair
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

We present a systematic approach for evaluating the quality of knowledge graph repairs with respect to constraint violations defined in shapes constraint language (SHACL). Current evaluation methods rely on emph{ad hoc} datasets, which limits the rigorous analysis of repair systems in more general settings. Our method addresses this gap by systematically generating violations using a novel mechanism, termed violation-inducing operations (VIOs). We use the proposed evaluation framework to assess a range of repair systems which we build using large language models. We analyze the performance of these systems across different prompting strategies. Results indicate that concise prompts containing both the relevant violated SHACL constraints and key contextual information from the knowledge graph yield the best performance.

Problem

• Evaluating knowledge graph repair quality systematically
• Addressing limitations of ad hoc evaluation datasets
• Assessing repair systems using large language models

Innovation

• Systematic violation generation using VIOs
• Evaluation framework for KG repair systems
• Optimal prompts with SHACL constraints