Adversarial Graph Neural Network Benchmarks: Towards Practical and Fair Evaluation

2026-05-06

AI Summary

This work addresses the lack of a unified evaluation standard in existing adversarial methods for graph neural networks (GNNs), which has led to ambiguous or even contradictory conclusions. We propose the first systematic evaluation framework, conducting over 450,000 fair and reproducible experiments across six graph datasets to re-evaluate seven attack and eight defense methods under both poisoning and evasion settings. Our study reveals, for the first time, the critical yet previously overlooked influence of factors such as target node selection and model training procedures on attack effectiveness. We further demonstrate that current evaluation protocols exhibit significant biases that substantially distort the perceived performance of these methods. These findings underscore the necessity of standardized evaluation practices to advance robust GNN research toward real-world applicability.

Abstract

Adversarial learning and the robustness of Graph Neural Networks (GNNs) are topics of widespread interest in the machine learning community, as documented by the number of adversarial attacks and defenses designed for these purposes. While a rigorous evaluation of these adversarial methods is necessary to understand the robustness of GNNs in real-world applications, we posit that many works in the literature do not share the same experimental settings, leading to ambiguous and potentially contradictory scientific conclusions. In this benchmark, we demonstrate the importance of adopting fair, robust, and standardized evaluation protocols in adversarial GNN research. We perform a comprehensive re-evaluation of seven widely used attacks and eight recent defenses under both poisoning and evasion scenarios, across six popular graph datasets. Our study spans over 453,000 experiments conducted within a unified framework. We observe substantial differences in adversarial attack performance when evaluated under a fair and robust procedure. Our findings reveal that previously overlooked factors, such as target node selection and the training process of the attacked model, have a profound impact on attack effectiveness, to the extent of completely distorting performance insights. These results underscore the urgent need for standardized evaluations in adversarial graph machine learning.

Problem

• Adversarial Graph Neural Networks
• Robustness Evaluation
• Standardized Benchmarking
• Fair Evaluation
• Graph Machine Learning

Innovation

• Adversarial Graph Neural Networks
• Benchmarking
• Fair Evaluation
• Robustness
• Standardized Protocol