Integrating Explainable AI for Effective Malware Detection in Encrypted Network Traffic

2025-01-09

AI Summary

Detecting unknown malware communications in encrypted network traffic remains challenging due to the inability to decrypt payloads and the absence of plaintext content for analysis. Method: This paper proposes a content-agnostic, interpretable detection framework. It introduces the largest publicly available encrypted malware traffic dataset to date (1,127 connections across 54 families), integrates multidimensional feature engineering—including packet length statistics, inter-arrival timing, and TLS version—and employs XGBoost for classification. Crucially, it is the first work to systematically apply SHAP for dual-granularity interpretability (global and local). Results: The method achieves >99% accuracy, precision, and F1-score on the CTU-13 dataset; on the proposed dataset, it attains 99.32% accuracy, 99.53% precision, and 99.43% F1-score. SHAP analysis identifies maximum packet length, mean inter-packet interval, and TLS version as the three most discriminative features.

Technology Category

• Machine Learning: Transparent, Interpretable, Explainable ML
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Data Mining & Knowledge Management: Representing, Reasoning, and Using Provenance, Trust

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Security and Privacy: Large-scale security measurements
• Responsible Web: Measurement, analysis, and circumvention of Web censorship

Abstract

Encrypted network communication ensures confidentiality, integrity, and privacy between endpoints. However, attackers are increasingly exploiting encryption to conceal malicious behavior. Detecting unknown encrypted malicious traffic without decrypting the payloads remains a significant challenge. In this study, we investigate the integration of explainable artificial intelligence (XAI) techniques to detect malicious network traffic. We employ ensemble learning models to identify malicious activity using multi-view features extracted from various aspects of encrypted communication. To effectively represent malicious communication, we compiled a robust dataset with 1,127 unique connections, more than any other available open-source dataset, and spanning 54 malware families. Our models were benchmarked against the CTU-13 dataset, achieving performance of over 99% accuracy, precision, and F1-score. Additionally, the eXtreme Gradient Boosting (XGB) model demonstrated 99.32% accuracy, 99.53% precision, and 99.43% F1-score on our custom dataset. By leveraging Shapley Additive Explanations (SHAP), we identified that the maximum packet size, mean inter-arrival time of packets, and transport layer security version used are the most critical features for the global model explanation. Furthermore, key features were identified as important for local explanations across both datasets for individual traffic samples. These insights provide a deeper understanding of the model decision-making process, enhancing the transparency and reliability of detecting malicious encrypted traffic.

Problem

• Encrypted Network Communication
• Malware Traffic Identification
• Content Encryption

Innovation

• Explainable AI (XAI)
• Encrypted Traffic Analysis
• Malware Detection