MLaGA: Multimodal Large Language and Graph Assistant

2025-06-03

AI Summary

Existing LLM-based graph methods are largely confined to text-rich graphs and struggle to model multimodal graphs containing heterogeneous node attributes (e.g., text and images). This paper introduces the first large language model framework specifically designed for multimodal graph-structured data. Our approach addresses the problem through three key innovations: (1) a structure-aware multimodal encoder that jointly encodes graph topology and cross-modal node attributes; (2) a joint pretraining objective that explicitly incorporates graph structural priors; and (3) a lightweight multimodal instruction fine-tuning mechanism with projection adapters. Evaluated on multiple multimodal graph benchmarks, our method achieves state-of-the-art performance in both supervised learning and cross-domain transfer tasks, significantly outperforming existing unimodal and multimodal graph models.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Large Language Models (LLMs) have demonstrated substantial efficacy in advancing graph-structured data analysis. Prevailing LLM-based graph methods excel in adapting LLMs to text-rich graphs, wherein node attributes are text descriptions. However, their applications to multimodal graphs--where nodes are associated with diverse attribute types, such as texts and images--remain underexplored, despite their ubiquity in real-world scenarios. To bridge the gap, we introduce the Multimodal Large Language and Graph Assistant (MLaGA), an innovative model that adeptly extends LLM capabilities to facilitate reasoning over complex graph structures and multimodal attributes. We first design a structure-aware multimodal encoder to align textual and visual attributes within a unified space through a joint graph pre-training objective. Subsequently, we implement a multimodal instruction-tuning approach to seamlessly integrate multimodal features and graph structures into the LLM through lightweight projectors. Extensive experiments across multiple datasets demonstrate the effectiveness of MLaGA compared to leading baseline methods, achieving superior performance in diverse graph learning tasks under both supervised and transfer learning scenarios.

Problem

• Extending LLMs to analyze multimodal graphs with diverse attributes
• Aligning textual and visual attributes in a unified graph space
• Integrating multimodal features and graph structures into LLMs effectively

Innovation

• Multimodal encoder aligns text and visuals
• Instruction-tuning integrates features into LLM
• Lightweight projectors enhance graph learning tasks