Multi-Level Distributional Entropy for Explainable Network Intrusion Detection

2026-06-29

AI Summary

This work addresses the limitations of existing network intrusion detection systems that rely on aggregated flow statistics, which discard distributional structure, and conventional entropy-based methods that require raw packets and are thus inapplicable to pre-aggregated data. The authors propose a Multilevel Distributional Entropy (MDE) framework that, for the first time, directly constructs interpretable, multilayer entropy features from flow-level summary statistics—specifically, intra-flow Gaussian differential entropy, inter-directional Jensen–Shannon divergence, and TCP flag Shannon entropy—without needing raw packets or training samples. Evaluated on four benchmark datasets, MDE alone achieves weighted F1 scores ranging from 0.708 to 0.989. SHAP analysis demonstrates high feature stability (Spearman ρ = 0.80–0.95) and reveals that fixed thresholds suffer severe performance degradation under temporal shifts, with detection rates plummeting to 0.082, thereby substantially enhancing detection transparency and robustness.

Abstract

Machine learning network intrusion detection systems (IDS) rely on aggregate flow statistics that discard distributional structure, while established entropy measures require raw packet sequences unavailable in pre-aggregated flow datasets. We propose Multi-Level Distributional Entropy (MDE), an analytical framework that derives interpretable entropy features directly from flow-level summary statistics at three levels: within-flow Gaussian differential entropy, cross-directional Jensen-Shannon divergence (JSD), and Transmission Control Protocol (TCP) flag-pattern Shannon entropy, without raw packet access or training data. Across four benchmarks (NSL-KDD, CICIDS-2017, CICIDS-2018, UNSW-NB15) under a leakage-free fold-local pipeline, entropy-only features achieve weighted F1 of 0.708-0.989, matching conventional features without degrading performance. Full operational metric reporting then exposes failure modes that aggregate F1 conceals. On CICIDS-2018, F1=0.74 hides a detection rate (DR) of 0.48, and on held-out attack families F1 exceeds 0.998 while DR falls to zero. Under temporal shift, a pseudo-live replay of 703K flows reveals a threshold-ranking divergence in which score ranking is preserved (AUC=0.87) but fixed thresholds collapse (DR=0.082) and recalibration offers no recovery. SHapley Additive exPlanations (SHAP) fold-stability analysis (Spearman rho=0.80-0.95) confirms that entropy attributions are reproducible and domain-coherent across heterogeneous environments.

Problem

• Network Intrusion Detection
• Distributional Entropy
• Flow-level Statistics
• Explainable AI
• Entropy Features

Innovation

• Multi-Level Distributional Entropy
• Explainable Intrusion Detection
• Flow-level Entropy Features
• Jensen-Shannon Divergence
• SHAP Stability