A Hard-Label Black-Box Evasion Attack against ML-based Malicious Traffic Detection Systems

2025-10-16

AI Summary

Existing evasion attacks against machine learning–based malicious traffic detection systems in hard-label black-box settings rely on strong assumptions or prior knowledge of the target model, limiting practical applicability. Method: We propose NetMasquerade—the first hard-label black-box evasion attack framework requiring no access to the target model’s architecture, parameters, or training data. It leverages Traffic-BERT, a pre-trained language model tailored for network traffic, to capture benign traffic semantics, and employs reinforcement learning to optimize adversarial traffic generation with minimal perturbation and high stealthiness. Contribution/Results: Evaluated across 80 cross-protocol and cross-task scenarios, NetMasquerade achieves an average attack success rate of 96.65% against six state-of-the-art detectors. It effectively bypasses both empirical and certified defenses, while maintaining low generation latency—demonstrating strong feasibility for real-world deployment.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Adversarial Attacks & Robustness
• Multiagent Systems: Adversarial Agents

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Responsible Web: Measurement, analysis, and circumvention of Web censorship
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Machine Learning (ML)-based malicious traffic detection is a promising security paradigm. It outperforms rule-based traditional detection by identifying various advanced attacks. However, the robustness of these ML models is largely unexplored, thereby allowing attackers to craft adversarial traffic examples that evade detection. Existing evasion attacks typically rely on overly restrictive conditions (e.g., encrypted protocols, Tor, or specialized setups), or require detailed prior knowledge of the target (e.g., training data and model parameters), which is impractical in realistic black-box scenarios. The feasibility of a hard-label black-box evasion attack (i.e., applicable across diverse tasks and protocols without internal target insights) thus remains an open challenge. To this end, we develop NetMasquerade, which leverages reinforcement learning (RL) to manipulate attack flows to mimic benign traffic and evade detection. Specifically, we establish a tailored pre-trained model called Traffic-BERT, utilizing a network-specialized tokenizer and an attention mechanism to extract diverse benign traffic patterns. Subsequently, we integrate Traffic-BERT into the RL framework, allowing NetMasquerade to effectively manipulate malicious packet sequences based on benign traffic patterns with minimal modifications. Experimental results demonstrate that NetMasquerade enables both brute-force and stealthy attacks to evade 6 existing detection methods under 80 attack scenarios, achieving over 96.65% attack success rate. Notably, it can evade the methods that are either empirically or certifiably robust against existing evasion attacks. Finally, NetMasquerade achieves low-latency adversarial traffic generation, demonstrating its practicality in real-world scenarios.

Problem

• Evaluating ML model robustness against hard-label black-box evasion attacks
• Developing practical adversarial traffic generation without target knowledge
• Creating attack method applicable across diverse protocols and tasks

Innovation

• Reinforcement learning manipulates attack flows
• Traffic-BERT extracts benign traffic patterns
• Minimal packet modifications evade detection methods