Adversarial Robustness in Smishing Detection: A Comparative Analysis of Adversarial Fragility in Classical vs. Transformer-Based Detection Systems

2026-08-13

AI Summary

This study addresses the insufficient robustness of existing smishing detection systems against adversarial attacks involving character obfuscation, cross-lingual mixing, and structural perturbations—particularly in low-resource settings where such attacks easily bypass detectors. The authors systematically evaluate three classical lexical models (Random Forest, XGBoost, CNN+BiLSTM) and two multilingual Transformer models (mBERT, XLM-RoBERTa) under various black-box and attention-guided attacks, quantifying performance degradation via the Robustness Degradation Ratio (RDR) and validating results with Mann-Whitney U and Friedman statistical tests. Their findings reveal, for the first time, that model architecture critically determines smishing detection robustness: classical models exhibit severe degradation (RDR up to 0.988), rendering them nearly ineffective, whereas Transformer-based models are significantly more resilient (maximum RDR of 0.351), with mBERT outperforming XLM-RoBERTa. Notably, clean accuracy proves an unreliable predictor of adversarial robustness, underscoring the need for architecture-specific defense mechanisms.

Abstract

Smishing detection systems are commonly trained and evaluated on clean, monolingual text. In low-resource settings, however, attackers frequently circumvent these systems through character obfuscation, cross-lingual code-switching, and structural perturbation. This study evaluates adversarial robustness for five model architectures: three classical lexical models (Random Forest, XGBoost, CNN+BiLSTM) and two multilingual transformers (mBERT, XLM-RoBERTa), using a dataset of 27,037 messages. Classical models are subjected to black-box generic attacks, while transformers are evaluated with attention-guided targeting. Each model is tested across three attack types and intensity levels, with performance measured by the Robustness Degradation Ratio (RDR). The results reveal a distinct architectural boundary: classical models experience near-catastrophic failure under character obfuscation and structural perturbation (RDR up to 0.988), whereas transformers demonstrate significantly greater resilience (RDR up to 0.351), with structural perturbation representing their most pronounced vulnerability. Effect-size analysis (Cliff's d) indicates a substantial difference between the two model categories. Within the transformer group, XLM-RoBERTa, despite achieving a higher clean-text baseline, exhibits greater degradation than mBERT. These findings demonstrate that clean-text performance is not a reliable predictor of adversarial robustness. Statistical validation using Mann-Whitney U and Friedman tests confirms that these patterns are attributable to model architecture rather than sampling. The results underscore the necessity for architecture-specific defences and frame smishing detection as an adversarial cybersecurity challenge rather than a static classification task.

Problem

• adversarial robustness
• smishing detection
• character obfusc日消息
• structural perturbation
• code-switching

Innovation

• adversarial robustness
• smishing detection
• transformer models
• robustness degradation ratio
• attention-guided attack