🤖 AI Summary
This study addresses the dual vulnerability of machine learning models in cybersecurity: adversarial attacks not only degrade prediction accuracy but also destabilize SHAP-based interpretability. Focusing on Random Forest and XGBoost classifiers across four tabular cybersecurity datasets, the authors propose an Explainability Stability Index (ESI) and integrate it with Robustness Index (RI) to form a two-dimensional evaluation framework leveraging black-box attack methods such as TreeSHAP, ZOO, and Square Attack. Their analysis reveals that while ZOO attacks yield deceptively high robustness scores on XGBoost (RI ≈ 0.98), they severely compromise explanation stability (ESI = 0.06–0.16). In contrast, Square Attack more faithfully exposes model fragility (RI ≈ 0.36), demonstrating that predictive robustness and explanation stability constitute distinct and independent dimensions of model reliability.
📝 Abstract
Adversarial attacks on cybersecurity classifiers pose a dual threat: degrading predictions and destabilising the SHAP-based explanations that security analysts rely on to understand and triage alerts. We extend our prior MLP conference study to Random Forest and XGBoost across four tabular security datasets (phishing URLs, UNSW-NB15, NF-ToN-IoT, HIKARI-2021), evaluating five attacks including three black-box methods applicable to non-differentiable tree models. We introduce the Explainability Stability Index (ESI), a scalar metric computed from TreeSHAP attribution drift under adversarial perturbation, reported on the same [0,1] scale as the Robustness Index (RI). A key finding is that gradient-based black-box attacks (ZOO) produce degenerate results against XGBoost (apparent RI ~0.98) due to piecewise-constant prediction surfaces, while score-based Square Attack reveals genuine vulnerability (RI ~0.36). These degenerate perturbations still drive substantial attribution drift: XGBoost ESI ~0.06-0.16 despite near-perfect ZOO robustness, versus 0.14-0.29 for RF, showing that prediction robustness and explanation stability are distinct axes requiring joint measurement. A two-axis framework (gradient dependence, query efficiency) explains the observed attack ranking and yields practical guidance for tree ensemble evaluation. A step-size ablation explains a counterintuitive PGD anomaly on z-score normalised tabular data.