Enhancing Knowledge Graph Construction: Evaluating with Emphasis on Hallucination, Omission, and Graph Similarity Metrics

2025-02-07

AI Summary

Large language models (LLMs) suffer from hallucination and entity/relation omission when automatically constructing knowledge graphs (KGs) from unstructured text. Method: We propose a graph-similarity evaluation framework integrating BERTScore, with 95% graph matching empirically established as a practical threshold. This is the first work to jointly quantify structural graph similarity, hallucination rate, and omission rate within a unified metric. We fine-tune Mistral on the KELM-sub dataset and evaluate KG generation performance using our framework. Contribution/Results: Our evaluation paradigm establishes the first end-to-end, fully quantifiable KG construction benchmark. Experiments show that fine-tuning significantly reduces precise hallucination and omission rates but exposes a critical trade-off between accuracy gain and degraded generalization. The 95% matching threshold is validated across realistic scenarios. Crucially, we systematically characterize the dual impact of LLM fine-tuning—enhancing factual precision while compromising structural robustness—in KG generation.

Technology Category

• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Knowledge Representation Languages

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Recent advancements in large language models have demonstrated significant potential in the automated construction of knowledge graphs from unstructured text. This paper builds upon our previous work [16], which evaluated various models using metrics like precision, recall, F1 score, triple matching, and graph matching, and introduces a refined approach to address the critical issues of hallucination and omission. We propose an enhanced evaluation framework incorporating BERTScore for graph similarity, setting a practical threshold of 95% for graph matching. Our experiments focus on the Mistral model, comparing its original and fine-tuned versions in zero-shot and few-shot settings. We further extend our experiments using examples from the KELM-sub training dataset, illustrating that the fine-tuned model significantly improves knowledge graph construction accuracy while reducing the exact hallucination and omission. However, our findings also reveal that the fine-tuned models perform worse in generalization tasks on the KELM-sub dataset. This study underscores the importance of comprehensive evaluation metrics in advancing the state-of-the-art in knowledge graph construction from textual data.

Problem

• Addresses hallucination and omission in knowledge graphs.
• Proposes enhanced evaluation using BERTScore for similarity.
• Tests Mistral model in zero-shot and few-shot settings.

Innovation

• BERTScore for graph similarity
• Mistral model fine-tuning
• 95% graph matching threshold