PsyScam: A Benchmark for Psychological Techniques in Real-World Scams

2025-05-21

AI Summary

Existing fraud benchmarks lack modeling of psychological tactics (PTs) employed in real-world scams. This paper introduces PsyScam—the first benchmark dataset grounded in authentic scam reports and rigorously informed by cognitive and psychological theories. It systematically curates cases from six major platforms and provides fine-grained human annotations. Methodologically, it pioneers the integration of empirically validated psychological theories into cybersecurity fraud analysis, enabling three core tasks: PT classification, scam text completion, and PT-aware text augmentation. An adversarial evaluation framework is also proposed. Experiments reveal substantial deficiencies in state-of-the-art large language models for both PT identification and generation. PsyScam fills a critical gap in interpretable PT modeling and evaluation, and all data and code are publicly released to advance interdisciplinary research at the intersection of explainable AI and anti-fraud technology.

Technology Category

• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction
• Application Domains: Humanities & Computational Social Science
• Natural Language Processing: Generation

Application Category

• User Modeling, Personalization and Recommendation: Psychology-informed user models and recommender systems
• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Online scams have become increasingly prevalent, with scammers using psychological techniques (PTs) to manipulate victims. While existing research has developed benchmarks to study scammer behaviors, these benchmarks do not adequately reflect the PTs observed in real-world scams. To fill this gap, we introduce PsyScam, a benchmark designed to systematically capture and evaluate PTs embedded in real-world scam reports. In particular, PsyScam bridges psychology and real-world cyber security analysis through collecting a wide range of scam reports from six public platforms and grounding its annotations in well-established cognitive and psychological theories. We further demonstrate PsyScam's utility through three downstream tasks: PT classification, scam completion, and scam augmentation. Experimental results show that PsyScam presents significant challenges to existing models in both detecting and generating scam content based on the PTs used by real-world scammers. Our code and dataset are available at: https://anonymous.4open.science/r/PsyScam-66E4.

Problem

• Lack of benchmarks for psychological techniques in scams
• Need to bridge psychology and cybersecurity in scam analysis
• Challenges in detecting and generating scam content accurately

Innovation

• Introduces PsyScam benchmark for real-world scam PTs
• Bridges psychology and cybersecurity via multi-platform reports
• Validates with PT classification and scam generation tasks