Hyper-modal Imputation Diffusion Embedding with Dual-Distillation for Federated Multimodal Knowledge Graph Completion

2025-06-27

AI Summary

To address the challenges of modality incompleteness and client heterogeneity in federated multimodal knowledge graph completion (FedMKGC) under cross-institutional data privatization, this paper proposes HidE-MMFeD3—a novel framework comprising two core components. First, the HidE model leverages hypermodal diffusion embedding to reconstruct incomplete multimodal distributions. Second, the MMFeD3 dual distillation mechanism jointly performs logit-level and feature-level federated distillation, ensuring semantic consistency between clients and server while accelerating global convergence. To enable rigorous evaluation, we introduce the first dedicated FedMKGC benchmark. Extensive experiments demonstrate that HidE-MMFeD3 significantly outperforms existing baselines in link prediction accuracy, semantic alignment fidelity, and convergence robustness—achieving effective privacy-preserving multimodal knowledge co-reasoning.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion
• Computer Vision: Multi-modal Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

With the increasing multimodal knowledge privatization requirements, multimodal knowledge graphs in different institutes are usually decentralized, lacking of effective collaboration system with both stronger reasoning ability and transmission safety guarantees. In this paper, we propose the Federated Multimodal Knowledge Graph Completion (FedMKGC) task, aiming at training over federated MKGs for better predicting the missing links in clients without sharing sensitive knowledge. We propose a framework named MMFeD3-HidE for addressing multimodal uncertain unavailability and multimodal client heterogeneity challenges of FedMKGC. (1) Inside the clients, our proposed Hyper-modal Imputation Diffusion Embedding model (HidE) recovers the complete multimodal distributions from incomplete entity embeddings constrained by available modalities. (2) Among clients, our proposed Multimodal FeDerated Dual Distillation (MMFeD3) transfers knowledge mutually between clients and the server with logit and feature distillation to improve both global convergence and semantic consistency. We propose a FedMKGC benchmark for a comprehensive evaluation, consisting of a general FedMKGC backbone named MMFedE, datasets with heterogeneous multimodal information, and three groups of constructed baselines. Experiments conducted on our benchmark validate the effectiveness, semantic consistency, and convergence robustness of MMFeD3-HidE.

Problem

• Decentralized multimodal knowledge graphs lack collaboration and safety
• Predict missing links in federated MKGs without sharing sensitive data
• Address multimodal uncertain unavailability and client heterogeneity challenges

Innovation

• Hyper-modal Imputation Diffusion Embedding model
• Multimodal Federated Dual Distillation technique
• Federated Multimodal Knowledge Graph benchmark