REAL-IoT: Characterizing GNN Intrusion Detection Robustness under Practical Adversarial Attack

2025-07-14

AI Summary

Existing GNN-based network intrusion detection systems (NIDS) suffer from two critical limitations: (1) evaluation predominantly relies on single benchmark datasets, hindering reliable assessment of generalization across heterogeneous environments; and (2) adversarial robustness is typically evaluated against non-physical, synthetically generated perturbations—overestimating real-world resilience. To address these issues, we propose REAL-IoT, the first framework to jointly model distributional shift and physically grounded adversarial attacks in IoT settings. It integrates a unified benchmark dataset comprising real attack traffic collected from a physical IoT testbed. Crucially, we introduce large language models (LLMs) to parse raw network flows and automatically filter anomalous samples, thereby enhancing feature fidelity. Experiments demonstrate that state-of-the-art GNN models exhibit substantial performance degradation under both realistic adversarial attacks and distribution shifts—validating their practical fragility—while LLM-assisted filtering significantly improves robustness. This work exposes systematic biases in conventional NIDS evaluation and establishes a new paradigm for trustworthy, operationally relevant NIDS assessment.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Adversarial Attacks & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Security and Privacy: Large-scale security measurements

Abstract

Graph Neural Network (GNN)-based network intrusion detection systems (NIDS) are often evaluated on single datasets, limiting their ability to generalize under distribution drift. Furthermore, their adversarial robustness is typically assessed using synthetic perturbations that lack realism. This measurement gap leads to an overestimation of GNN-based NIDS resilience. To address the limitations, we propose extbf{REAL-IoT}, a comprehensive framework for robustness evaluation of GNN-based NIDS in IoT environments. Our framework presents a methodology that creates a unified dataset from canonical datasets to assess generalization under drift. In addition, it features a novel intrusion dataset collected from a physical IoT testbed, which captures network traffic and attack scenarios under real-world settings. Furthermore, using REAL-IoT, we explore the usage of Large Language Models (LLMs) to analyze network data and mitigate the impact of adversarial examples by filtering suspicious flows. Our evaluations using REAL-IoT reveal performance drops in GNN models compared to results from standard benchmarks, quantifying their susceptibility to drift and realistic attacks. We also demonstrate the potential of LLM-based filtering to enhance robustness. These findings emphasize the necessity of realistic threat modeling and rigorous measurement practices for developing resilient IoT intrusion detection systems.

Problem

• Assessing GNN-based NIDS generalization under distribution drift
• Evaluating adversarial robustness with realistic attack scenarios
• Exploring LLMs for mitigating adversarial impacts in IoT security

Innovation

• Unified dataset for GNN generalization assessment
• Physical IoT testbed for realistic attack data
• LLM-based filtering to counter adversarial examples