Scam Shield: Multi-Model Voting and Fine-Tuned LLMs Against Adversarial Attacks

2025-11-03

AI Summary

To address insufficient robustness of fraud detection systems against adversarial evasion in scam messages, this paper proposes a hierarchical fraud detection framework. Methodologically: (1) a lightweight four-classifier ensemble with majority voting performs coarse-grained filtering; (2) a fine-grained detection module is built upon LLaMA-3.1-8B-Instruct and enhanced via adversarial training to improve resilience against perturbed inputs; (3) a dynamic inference routing strategy balances accuracy and efficiency by adaptively assigning tasks across modules. Experimental results demonstrate that the system significantly outperforms conventional machine learning models and commercial large language model baselines in adversarial sample detection—achieving an F1-score of 0.92 while reducing inference latency by 37%. The framework thus establishes a deployable, robust paradigm for real-time content moderation in security-critical applications.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness
• Multiagent Systems: Adversarial Agents

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Web Mining and Content Analysis: Large pretrained models with web data
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Scam detection remains a critical challenge in cybersecurity as adversaries craft messages that evade automated filters. We propose a Hierarchical Scam Detection System (HSDS) that combines a lightweight multi-model voting front end with a fine-tuned LLaMA 3.1 8B Instruct back end to improve accuracy and robustness against adversarial attacks. An ensemble of four classifiers provides preliminary predictions through majority vote, and ambiguous cases are escalated to the fine-tuned model, which is optimized with adversarial training to reduce misclassification. Experiments show that this hierarchical design both improves adversarial scam detection and shortens inference time by routing most cases away from the LLM, outperforming traditional machine-learning baselines and proprietary LLM baselines. The findings highlight the effectiveness of a hybrid voting mechanism and adversarial fine-tuning in fortifying LLMs against evolving scam tactics, enhancing the resilience of automated scam detection systems.

Problem

• Detecting adversarial scam messages evading automated filters
• Improving accuracy and robustness against evolving scam tactics
• Reducing misclassification while maintaining efficient inference time

Innovation

• Hierarchical system combines voting ensemble with fine-tuned LLM
• Multi-model voting front end routes cases for efficiency
• Adversarial fine-tuning optimizes LLM for scam detection