MalMoE: Mixture-of-Experts Enhanced Encrypted Malicious Traffic Detection Under Graph Drift

2026-02-10

AI Summary

This work addresses the challenge of effectively detecting malicious behavior in encrypted network traffic, which is hindered by opaque payloads and temporal graph structure drift. To this end, the authors propose MalMoE, a novel system that models multi-host interactions via graphs and employs a set of 1-hop Graph Neural Network (GNN) expert modules tailored to different drift patterns. MalMoE introduces a drift-aware gating mechanism that dynamically routes inputs to the most suitable experts based on observed structural shifts. The system further enhances model stability through a two-stage training strategy combined with data augmentation. Experimental results demonstrate that MalMoE achieves high-accuracy, real-time malicious traffic detection across open-source, synthetic, and real-world datasets, significantly mitigating performance degradation caused by graph drift.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Data Mining & Knowledge Management: Graph Mining, Social Network Analysis & Community
• Computer Vision: Multi-modal Vision

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Encryption has been commonly used in network traffic to secure transmission, but it also brings challenges for malicious traffic detection, due to the invisibility of the packet payload. Graph-based methods are emerging as promising solutions by leveraging multi-host interactions to promote detection accuracy. But most of them face a critical problem: Graph Drift, where the flow statistics or topological information of a graph change over time. To overcome these drawbacks, we propose a graph-assisted encrypted traffic detection system, MalMoE, which applies Mixture of Experts (MoE) to select the best expert model for drift-aware classification. Particularly, we design 1-hop-GNN-like expert models that handle different graph drifts by analyzing graphs with different features. Then, the redesigned gate model conducts expert selection according to the actual drift. MalMoE is trained with a stable two-stage training strategy with data augmentation, which effectively guides the gate on how to perform routing. Experiments on open-source, synthetic, and real-world datasets show that MalMoE can perform precise and real-time detection.

Problem

• encrypted traffic
• malicious traffic detection
• graph drift
• network security
• payload invisibility

Innovation

• Mixture of Experts
• Graph Drift
• Encrypted Traffic Detection
• Graph Neural Network
• Dynamic Routing