PhishIntentionLLM: Uncovering Phishing Website Intentions through Multi-Agent Retrieval-Augmented Generation

2025-07-21

AI Summary

This paper addresses the underexplored problem of malicious intent identification in phishing websites. Methodologically, it introduces the first intent-aware phishing analysis framework: (1) systematically defines and annotates four primary phishing intent categories, constructing a large-scale, real-world intent dataset comprising 9K samples; (2) proposes a multi-agent retrieval-augmented generation (RAG) architecture that integrates vision-language foundation models (e.g., GPT-4o) with collaborative multi-agent reasoning to enable fine-grained intent classification and interpretable inference. Experimental results demonstrate that the framework achieves a micro-precision of 0.7895 on GPT-4o—representing a 95% improvement over single-agent baselines—and attains 0.8545 precision for credential-theft intent. Key contributions include: (i) the first taxonomy for phishing intent classification; (ii) the first large-scale, human-annotated phishing intent dataset; and (iii) the first multi-agent RAG framework specifically designed for phishing intent recognition.

Technology Category

• Multiagent Systems: Adversarial Agents
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Phishing websites remain a major cybersecurity threat, yet existing methods primarily focus on detection, while the recognition of underlying malicious intentions remains largely unexplored. To address this gap, we propose PhishIntentionLLM, a multi-agent retrieval-augmented generation (RAG) framework that uncovers phishing intentions from website screenshots. Leveraging the visual-language capabilities of large language models (LLMs), our framework identifies four key phishing objectives: Credential Theft, Financial Fraud, Malware Distribution, and Personal Information Harvesting. We construct and release the first phishing intention ground truth dataset (~2K samples) and evaluate the framework using four commercial LLMs. Experimental results show that PhishIntentionLLM achieves a micro-precision of 0.7895 with GPT-4o and significantly outperforms the single-agent baseline with a ~95% improvement in micro-precision. Compared to the previous work, it achieves 0.8545 precision for credential theft, marking a ~4% improvement. Additionally, we generate a larger dataset of ~9K samples for large-scale phishing intention profiling across sectors. This work provides a scalable and interpretable solution for intention-aware phishing analysis.

Problem

• Uncovering phishing website malicious intentions from screenshots
• Identifying key phishing objectives using visual-language LLMs
• Providing scalable solution for intention-aware phishing analysis

Innovation

• Multi-agent RAG framework for phishing intentions
• Leverages visual-language LLM capabilities
• First phishing intention dataset with 2K samples