PhishDebate: An LLM-Based Multi-Agent Framework for Phishing Website Detection

2025-06-18

AI Summary

Phishing website detection faces critical challenges including hallucination, poor interpretability, and insufficient robustness. To address these, we propose the first multi-agent large language model (LLM) debate framework tailored for this task. It comprises four specialized agents—URL analyzer, HTML parser, semantic content interpreter, and brand impersonation detector—that collaboratively perform structured deliberation and divergence reasoning under the coordination of a moderator and final adjudication by a judge. The framework supports modular configuration and dynamic adaptation, integrating web parsing, semantic similarity comparison, brand knowledge retrieval, and debate-driven reasoning chains. Evaluated on real-world datasets, it achieves 98.2% recall and 98.2% true positive rate—substantially outperforming single-agent and chain-of-thought (CoT) baselines—while delivering high accuracy, strong interpretability, and enhanced hallucination resistance.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Machine Learning: Large Multimodal Models (LMMs)
• Multiagent Systems: Adversarial Agents

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Phishing websites continue to pose a significant cybersecurity threat, often leveraging deceptive structures, brand impersonation, and social engineering tactics to evade detection. While recent advances in large language models (LLMs) have enabled improved phishing detection through contextual understanding, most existing approaches rely on single-agent classification facing the risks of hallucination and lack interpretability or robustness. To address these limitations, we propose PhishDebate, a modular multi-agent LLM-based debate framework for phishing website detection. PhishDebate employs four specialized agents to independently analyze different textual aspects of a webpage--URL structure, HTML composition, semantic content, and brand impersonation--under the coordination of a Moderator and a final Judge. Through structured debate and divergent thinking, the framework delivers more accurate and interpretable decisions. Extensive evaluations on commercial LLMs demonstrate that PhishDebate achieves 98.2% recall and 98.2% True Positive Rate (TPR) on a real-world phishing dataset, and outperforms single-agent and Chain of Thought (CoT) baselines. Additionally, its modular design allows agent-level configurability, enabling adaptation to varying resource and application requirements.

Problem

• Detect phishing websites using multi-agent LLM framework
• Improve accuracy and interpretability in phishing detection
• Address single-agent limitations like hallucination and robustness

Innovation

• Multi-agent LLM framework for phishing detection
• Specialized agents analyze diverse webpage aspects
• Modular design enables configurable agent adaptation