ConceptFormer: Towards Efficient Use of Knowledge-Graph Embeddings in Large Language Models

2025-04-10

AI Summary

How can structured knowledge be efficiently injected into large language models (LLMs) to enhance factual recall—without modifying the LLM architecture or textualizing knowledge graphs (KGs)? This paper proposes ConceptFormer: a method that maps KG nodes (e.g., Wikidata entities) directly onto queryable concept vectors within the frozen LLM embedding space, enabling lightweight vector-based lookup. ConceptFormer integrates KG embedding, co-training with a frozen LLM, concept vector generation, and cross-space alignment—yielding a “one-vector, plug-and-play” knowledge enhancement mechanism. Evaluated on GPT-2 (0.1B), it achieves up to a 348% improvement in Hit@10 factual recall; injecting just one concept vector boosts recall by 213%, while reducing input token overhead by 130× compared to baseline prompting.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Retrieval Augmented Generation (RAG) has enjoyed increased attention in the recent past and recent advancements in Large Language Models (LLMs) have highlighted the importance of integrating world knowledge into these systems. Current RAG methodologies often modify the internal architecture of pre-trained language models (PLMs) or rely on textifying knowledge graphs (KGs), which is inefficient in terms of token usage. This paper introduces ConceptFormer, a new approach to augment LLMs with structured knowledge from KGs, such as Wikidata, without altering their internal structure or relying on textual input of KGs. ConceptFormer operates in the LLM embedding vector space, creating and injecting emph{concept vectors} that encapsulate the information of the KG nodes directly. Trained in conjunction with a frozen LLM, ConceptFormer generates a comprehensive lookup table that maps KG nodes to their respective concept vectors. The approach aims to enhance the factual recall capabilities of LLMs by enabling them to process these concept vectors natively, thus enriching them with structured world knowledge in an efficient and scalable manner. Our experiments demonstrate that the addition of concept vectors to GPT-2 0.1B substantially increases its factual recall ability (Hit@10) by up to 272% when tested on sentences from Wikipedia and up to 348% on synthetically generated sentences. Even injecting only a single concept vector into the prompt increases factual recall ability (Hit@10) by up to 213% on Wikipedia sentences, significantly outperforming RAG with graph textification while consuming 130x fewer input tokens.

Problem

• Efficiently integrating knowledge-graph embeddings into LLMs without architectural changes
• Reducing token usage by avoiding textual input of knowledge graphs
• Enhancing factual recall in LLMs using structured concept vectors

Innovation

• Uses concept vectors in embedding space
• Generates lookup table for KG nodes
• Enhances LLMs without internal modifications