Croppable Knowledge Graph Embedding

2024-07-03

arXiv.org

AI Summary

Fixed-dimensional knowledge graph embedding (KGE) models suffer from high training costs and poor flexibility, as adapting to new dimensionalities necessitates full retraining. This paper proposes MED, the first framework enabling *one-time training* of a *prunable* KGE model: arbitrary lower-dimensional submodels can be extracted on demand and deployed immediately—without retraining. Its core innovations include (i) mutual learning for joint high- and low-dimensional training, (ii) an evolutionary parameter optimization mechanism, (iii) a dynamically weighted multi-objective loss function, and (iv) a dimension-decoupled architecture ensuring effective knowledge transfer across dimensionalities. Extensive experiments on four knowledge graph completion benchmarks, three real-world applications, and a BERT extension task demonstrate that MED significantly reduces training overhead while maintaining—or even surpassing—the performance of full-dimensional models.

Technology Category

• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion
• Machine Learning: Graph-based Machine Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Knowledge Graph Embedding (KGE) is a common method for Knowledge Graphs (KGs) to serve various artificial intelligence tasks. The suitable dimensions of the embeddings depend on the storage and computing conditions of the specific application scenarios. Once a new dimension is required, a new KGE model needs to be trained from scratch, which greatly increases the training cost and limits the efficiency and flexibility of KGE in serving various scenarios. In this work, we propose a novel KGE training framework MED, through which we could train once to get a croppable KGE model applicable to multiple scenarios with different dimensional requirements, sub-models of the required dimensions can be cropped out of it and used directly without any additional training. In MED, we propose a mutual learning mechanism to improve the low-dimensional sub-models performance and make the high-dimensional sub-models retain the capacity that low-dimensional sub-models have, an evolutionary improvement mechanism to promote the high-dimensional sub-models to master the knowledge that the low-dimensional sub-models can not learn, and a dynamic loss weight to balance the multiple losses adaptively. Experiments on 3 KGE models over 4 standard KG completion datasets, 3 real application scenarios over a real-world large-scale KG, and the experiments of extending MED to the language model BERT show the effectiveness, high efficiency, and flexible extensibility of MED.

Problem

• Enables single training for multi-dimensional KG embedding models
• Allows cropping sub-models without retraining for different dimensions
• Improves low and high-dimensional sub-model performance simultaneously

Innovation

• Croppable KGE model for multiple dimensions
• Mutual learning mechanism enhances sub-models
• Dynamic loss weight balances multiple losses