Representation Learning with Mutual Influence of Modalities for Node Classification in Multi-Modal Heterogeneous Networks

2025-05-12

AI Summary

Existing multimodal fusion methods for node classification in multimodal heterogeneous networks (MMHNs) struggle to simultaneously preserve unimodal features and enable cross-modal collaborative guidance. Method: We propose HGNN-IMA, a heterogeneous graph neural network built upon a heterogeneous graph Transformer architecture. It introduces a nested cross-modal attention mechanism to jointly model intra-modal structural relationships and inter-modal dynamic mutual enhancement, augmented by modality alignment constraints and an attention-based loss to improve robustness under modality missingness. Contribution/Results: HGNN-IMA is the first method to achieve adaptive multimodal fusion and fine-grained cross-modal alignment during information propagation. Experiments on multiple real-world MMHN benchmarks demonstrate an average 3.2% improvement in node classification accuracy over state-of-the-art methods, with significantly enhanced stability under partial modality absence.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Nowadays, numerous online platforms can be described as multi-modal heterogeneous networks (MMHNs), such as Douban's movie networks and Amazon's product review networks. Accurately categorizing nodes within these networks is crucial for analyzing the corresponding entities, which requires effective representation learning on nodes. However, existing multi-modal fusion methods often adopt either early fusion strategies which may lose the unique characteristics of individual modalities, or late fusion approaches overlooking the cross-modal guidance in GNN-based information propagation. In this paper, we propose a novel model for node classification in MMHNs, named Heterogeneous Graph Neural Network with Inter-Modal Attention (HGNN-IMA). It learns node representations by capturing the mutual influence of multiple modalities during the information propagation process, within the framework of heterogeneous graph transformer. Specifically, a nested inter-modal attention mechanism is integrated into the inter-node attention to achieve adaptive multi-modal fusion, and modality alignment is also taken into account to encourage the propagation among nodes with consistent similarities across all modalities. Moreover, an attention loss is augmented to mitigate the impact of missing modalities. Extensive experiments validate the superiority of the model in the node classification task, providing an innovative view to handle multi-modal data, especially when accompanied with network structures.

Problem

• Node classification in multi-modal heterogeneous networks (MMHNs)
• Capturing mutual influence of modalities during information propagation
• Handling missing modalities and modality alignment in representation learning

Innovation

• Inter-modal attention for adaptive multi-modal fusion
• Modality alignment for consistent similarity propagation
• Attention loss to mitigate missing modalities impact