Large Language Model Enhanced Knowledge Representation Learning: A Survey

2024-07-01

arXiv.org

AI Summary

The severe sparsity of knowledge graphs (KGs) critically hinders the performance of knowledge representation learning (KRL) in downstream tasks such as link prediction, question answering, and logical reasoning. Method: This paper systematically surveys recent advances in large language model (LLM)-enhanced KRL and proposes, for the first time, a three-fold paradigm taxonomy—encoder-driven, encoder-decoder-cooperative, and decoder-dominant—alongside a task-adaptive knowledge-enhancement framework. The framework integrates Transformer architectures, KG embedding techniques, sequence-to-sequence modeling, context-aware encoding, and large-scale semantic decoding. Contribution/Results: Experimental results demonstrate that the proposed framework significantly improves the generalization and robustness of KRL models while alleviating the KG sparsity bottleneck. It establishes the first unified, scalable theoretical and practical framework for synergistic KG–LLM modeling, enabling principled integration of symbolic knowledge and neural language understanding.

Technology Category

• Knowledge Representation and Reasoning: Knowledge Representation Languages
• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Knowledge Representation Learning (KRL) is crucial for enabling applications of symbolic knowledge from Knowledge Graphs (KGs) to downstream tasks by projecting knowledge facts into vector spaces. Despite their effectiveness in modeling KG structural information, KRL methods are suffering from the sparseness of KGs. The rise of Large Language Models (LLMs) built on the Transformer architecture present promising opportunities for enhancing KRL by incorporating textual information to address information sparsity in KGs. LLM-enhanced KRL methods, including three key approaches, encoder-based methods that leverage detailed contextual information, encoder-decoder-based methods that utilize a unified seq2seq model for comprehensive encoding and decoding, and decoder-based methods that utilize extensive knowledge from large corpora, has significantly advanced the effectiveness and generalization of KRL in addressing a wide range of downstream tasks. This work provides a broad overview of downstream tasks while simultaneously identifying emerging research directions in these evolving domains.

Problem

• Enhance Knowledge Representation Learning
• Address KG information sparsity
• Integrate LLMs for KRL improvement

Innovation

• LLM-enhanced KRL methods
• Incorporating textual information
• Addressing KG sparsity