IGL-Bench: Establishing the Comprehensive Benchmark for Imbalanced Graph Learning

2024-06-14

arXiv.org

AI Summary

This work addresses the lack of standardized evaluation protocols in imbalanced graph learning (IGL). We introduce the first comprehensive IGL benchmark, comprising 16 graph datasets and 24 algorithms, and pioneer the systematic joint evaluation of both class imbalance and topological imbalance—two distinct yet interrelated challenges. To ensure reproducibility and fairness, we propose standardized preprocessing, data splitting, and evaluation protocols; define multi-granularity metrics (F1-macro, AUC-ROC, inference latency); and release an open-source, end-to-end evaluation toolkit. Extensive experiments reveal widespread robustness deficiencies of existing IGL methods under topological imbalance. Our benchmark establishes authoritative baselines and diagnostic tools, enabling rigorous comparative analysis. It has been widely adopted by the research community, significantly advancing fair evaluation and methodological progress in IGL.

Technology Category

• Machine Learning: Graph-based Machine Learning
• Data Mining & Knowledge Management: Graph Mining, Social Network Analysis & Community
• Reasoning under Uncertainty: Graphical Models

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Deep graph learning has gained grand popularity over the past years due to its versatility and success in representing graph data across a wide range of domains. However, the pervasive issue of imbalanced graph data distributions, where certain parts exhibit disproportionally abundant data while others remain sparse, undermines the efficacy of conventional graph learning algorithms, leading to biased outcomes. To address this challenge, Imbalanced Graph Learning (IGL) has garnered substantial attention, enabling more balanced data distributions and better task performance. Despite the proliferation of IGL algorithms, the absence of consistent experimental protocols and fair performance comparisons pose a significant barrier to comprehending advancements in this field. To bridge this gap, we introduce IGL-Bench, a foundational comprehensive benchmark for imbalanced graph learning, embarking on 16 diverse graph datasets and 24 distinct IGL algorithms with uniform data processing and splitting strategies. Specifically, IGL-Bench systematically investigates state-of-the-art IGL algorithms in terms of effectiveness, robustness, and efficiency on node-level and graph-level tasks, with the scope of class-imbalance and topology-imbalance. Extensive experiments demonstrate the potential benefits of IGL algorithms on various imbalanced conditions, offering insights and opportunities in the IGL field. Further, we have developed an open-sourced and unified package to facilitate reproducible evaluation and inspire further innovative research, which is available at https://github.com/RingBDStack/IGL-Bench.

Problem

• Addresses imbalanced graph data distribution challenges.
• Establishes a benchmark for evaluating IGL algorithms.
• Provides a unified framework for reproducible IGL research.

Innovation

• IGL-Bench benchmark for imbalanced graph learning
• Uniform data processing and splitting strategies
• Open-sourced package for reproducible evaluation