OpenMAG: A Comprehensive Benchmark for Multimodal-Attributed Graph

2026-02-05

AI Summary

This work addresses the lack of a unified and comprehensive evaluation benchmark for multimodal attribute graph (MAG) models, which has led to significant limitations in domain coverage, encoder flexibility, model diversity, and task scope. To bridge this gap, we propose a standardized benchmark framework encompassing 19 datasets across six domains, 16 encoders, 24 state-of-the-art models, and eight downstream tasks. Our framework supports both static and trainable feature encoders and integrates a curated model library, multimodal encoders, graph neural networks, and a unified training and evaluation pipeline. It enables, for the first time, systematic integration of cross-domain multimodal graph data and diverse architectures, achieving advances in encoder flexibility, task breadth, and evaluation dimensions. Through systematic assessment across five criteria—necessity, data quality, effectiveness, robustness, and efficiency—we derive 14 key insights, offering the community a reproducible, reliable benchmark and empirical guidance.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Multimodal-Attributed Graph (MAG) learning has achieved remarkable success in modeling complex real-world systems by integrating graph topology with rich attributes from multiple modalities. With the rapid proliferation of novel MAG models capable of handling intricate cross-modal semantics and structural dependencies, establishing a rigorous and unified evaluation standard has become imperative. Although existing benchmarks have facilitated initial progress, they exhibit critical limitations in domain coverage, encoder flexibility, model diversity, and task scope, presenting significant challenges to fair evaluation. To bridge this gap, we present OpenMAG, a comprehensive benchmark that integrates 19 datasets across 6 domains and incorporates 16 encoders to support both static and trainable feature encoding. OpenMAG further implements a standardized library of 24 state-of-the-art models and supports 8 downstream tasks, enabling fair comparisons within a unified framework. Through systematic assessment of necessity, data quality, effectiveness, robustness, and efficiency, we derive 14 fundamental insights into MAG learning to guide future advancements. Our code is available at https://github.com/YUKI-N810/OpenMAG.

Problem

• Multimodal-Attributed Graph
• benchmark
• evaluation standard
• model comparison
• graph learning

Innovation

• Multimodal-Attributed Graph
• comprehensive benchmark
• unified evaluation framework
• cross-modal semantics
• graph representation learning