SpaLLM-Guard: Pairing SMS Spam Detection Using Open-source and Commercial LLMs

2025-01-09

AI Summary

SMS spam threats exhibit dynamic evolution and heterogeneous strategies, posing significant challenges to robust detection. Method: This paper systematically evaluates mainstream open-source and commercial large language models—including GPT-4, DeepSeek, LLaMA-2, and Mixtral—under zero-shot, few-shot, chain-of-thought prompting, and supervised fine-tuning settings. Contribution/Results: We first identify supervised fine-tuning as the critical pathway to enhancing robustness against concept drift and adversarial attacks, and propose a synergistic “fine-tuning + few-shot” framework. Experiments demonstrate that fine-tuned Mixtral achieves 98.6% accuracy with both false positive and false negative rates below 2%. It exhibits strong robustness against both perceptible and imperceptible adversarial perturbations and maintains consistently high performance across temporally evolving spam datasets.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

The increasing threat of SMS spam, driven by evolving adversarial techniques and concept drift, calls for more robust and adaptive detection methods. In this paper, we evaluate the potential of large language models (LLMs), both open-source and commercial, for SMS spam detection, comparing their performance across zero-shot, few-shot, fine-tuning, and chain-of-thought prompting approaches. Using a comprehensive dataset of SMS messages, we assess the spam detection capabilities of prominent LLMs such as GPT-4, DeepSeek, LLAMA-2, and Mixtral. Our findings reveal that while zero-shot learning provides convenience, it is unreliable for effective spam detection. Few-shot learning, particularly with carefully selected examples, improves detection but exhibits variability across models. Fine-tuning emerges as the most effective strategy, with Mixtral achieving 98.6% accuracy and a balanced false positive and false negative rate below 2%, meeting the criteria for robust spam detection. Furthermore, we explore the resilience of these models to adversarial attacks, finding that fine-tuning significantly enhances robustness against both perceptible and imperceptible manipulations. Lastly, we investigate the impact of concept drift and demonstrate that fine-tuned LLMs, especially when combined with few-shot learning, can mitigate its effects, maintaining high performance even on evolving spam datasets. This study highlights the importance of fine-tuning and tailored learning strategies to deploy LLMs effectively for real-world SMS spam detection

Problem

• Spam SMS Detection
• Large Language Models
• Evolving Strategies

Innovation

• Large Language Models
• Specialized Training
• Few-shot Learning