Training Diverse Graph Experts for Ensembles: A Systematic Empirical Study

2025-10-21

AI Summary

This work addresses the heterogeneity inherent in real-world graph data by systematically investigating the impact of expert-level diversification strategies on Graph Neural Network (GNN) ensemble performance. We empirically evaluate 20 distinct diversification methods—including random reinitialization, architectural variation, directional modeling, data partitioning, and hyperparameter perturbation—across 14 node classification benchmarks, delivering the first comprehensive quantitative analysis of GNN expert diversity. Leveraging a Mixture-of-Experts framework, we construct and evaluate over 200 ensemble variants, uncovering a nonlinear relationship between diversity, complementarity, and generalization performance. We further propose a reproducible, principled training guideline for diversified GNN ensembles. Results demonstrate that judiciously introduced expert diversity significantly enhances both robustness and accuracy, with top-performing ensembles outperforming their best single-model baselines by up to 5.2% (average improvement). The implementation is publicly available.

Technology Category

• Machine Learning: Ensemble Methods
• Data Mining & Knowledge Management: Graph Mining, Social Network Analysis & Community
• Reasoning under Uncertainty: Graphical Models

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Graph Neural Networks (GNNs) have become essential tools for learning on relational data, yet the performance of a single GNN is often limited by the heterogeneity present in real-world graphs. Recent advances in Mixture-of-Experts (MoE) frameworks demonstrate that assembling multiple, explicitly diverse GNNs with distinct generalization patterns can significantly improve performance. In this work, we present the first systematic empirical study of expert-level diversification techniques for GNN ensembles. Evaluating 20 diversification strategies -- including random re-initialization, hyperparameter tuning, architectural variation, directionality modeling, and training data partitioning -- across 14 node classification benchmarks, we construct and analyze over 200 ensemble variants. Our comprehensive evaluation examines each technique in terms of expert diversity, complementarity, and ensemble performance. We also uncovers mechanistic insights into training maximally diverse experts. These findings provide actionable guidance for expert training and the design of effective MoE frameworks on graph data. Our code is available at https://github.com/Hydrapse/bench-gnn-diversification.

Problem

• Systematically evaluates diverse GNN expert training for ensemble performance
• Investigates 20 diversification strategies across 14 node classification benchmarks
• Provides mechanistic insights for training maximally complementary graph experts

Innovation

• Systematically evaluates 20 GNN diversification strategies
• Constructs 200 ensemble variants across 14 benchmarks
• Provides mechanistic insights for training diverse experts