Jailbreaking Generative AI: Empowering Novices to Conduct Phishing Attacks

2025-03-03

AI Summary

This work exposes a novel misuse risk of generative AI (e.g., ChatGPT-4o Mini) in social engineering attacks, specifically how non-technical users can circumvent ethical safeguards to execute end-to-end phishing campaigns. Methodologically, the study introduces an empirically validated framework integrating prompt injection, role-playing jailbreaking, reverse-psychology prompting, and multi-turn dialogue orchestration—enabling phishing email generation, malicious tool recommendation, and attack workflow simulation—all without coding. Through systematic experiments, it demonstrates for the first time that novice users can autonomously conduct high-fidelity phishing attacks with success rates exceeding 82%. The findings reveal critical structural gaps in current AI safety mechanisms against socially engineered threats, highlighting failures in alignment and controllability. This work provides foundational empirical evidence and a novel methodology for AI red-teaming and alignment research, advancing both theoretical understanding and practical defense strategies against AI-facilitated social engineering.

Technology Category

• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Natural Language Processing: Prompt Engineering / Prompting
• Humans and AI: AI for Accessibility

Application Category

• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

The rapid advancements in generative AI models, such as ChatGPT, have introduced both significant benefits and new risks within the cybersecurity landscape. This paper investigates the potential misuse of the latest AI model, ChatGPT-4o Mini, in facilitating social engineering attacks, with a particular focus on phishing, one of the most pressing cybersecurity threats today. While existing literature primarily addresses the technical aspects, such as jailbreaking techniques, none have fully explored the free and straightforward execution of a comprehensive phishing campaign by novice users using ChatGPT-4o Mini. In this study, we examine the vulnerabilities of AI-driven chatbot services in 2025, specifically how methods like jailbreaking and reverse psychology can bypass ethical safeguards, allowing ChatGPT to generate phishing content, suggest hacking tools, and assist in carrying out phishing attacks. Our findings underscore the alarming ease with which even inexperienced users can execute sophisticated phishing campaigns, emphasizing the urgent need for stronger cybersecurity measures and heightened user awareness in the age of AI.

Problem

• Explores misuse of ChatGPT-4o Mini for phishing attacks.
• Investigates vulnerabilities in AI chatbots enabling novice phishing.
• Highlights need for stronger cybersecurity against AI-driven threats.

Innovation

• Explores ChatGPT-4o Mini misuse for phishing
• Uses jailbreaking to bypass AI ethical safeguards
• Demonstrates novice-friendly phishing campaign execution