PhreshPhish: A Real-World, High-Quality, Large-Scale Phishing Website Dataset and Benchmark

2025-07-14

AI Summary

Existing phishing website detection research suffers from small-scale, low-quality datasets plagued by label noise, data leakage, and base-rate distortion—leading to overly optimistic model evaluations. To address these issues, we introduce PhishBench, the first large-scale, high-quality phishing dataset. It is constructed via rigorous data cleaning, manual verification, and real-world traffic sampling to eliminate label errors and leakage, while calibrating the realistic phishing base rate (~1%). We further propose a multi-difficulty benchmarking protocol covering cross-domain generalization, zero-shot transfer, and adversarial robustness. All data, code, and a unified evaluation framework are publicly released on Hugging Face, along with comprehensive results from multiple state-of-the-art baselines. PhishBench establishes the first standardized, reproducible evaluation benchmark for phishing detection research.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness

Application Category

• Web Mining and Content Analysis: Web data provenance, reliability, and authenticity
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

Phishing remains a pervasive and growing threat, inflicting heavy economic and reputational damage. While machine learning has been effective in real-time detection of phishing attacks, progress is hindered by lack of large, high-quality datasets and benchmarks. In addition to poor-quality due to challenges in data collection, existing datasets suffer from leakage and unrealistic base rates, leading to overly optimistic performance results. In this paper, we introduce PhreshPhish, a large-scale, high-quality dataset of phishing websites that addresses these limitations. Compared to existing public datasets, PhreshPhish is substantially larger and provides significantly higher quality, as measured by the estimated rate of invalid or mislabeled data points. Additionally, we propose a comprehensive suite of benchmark datasets specifically designed for realistic model evaluation by minimizing leakage, increasing task difficulty, enhancing dataset diversity, and adjustment of base rates more likely to be seen in the real world. We train and evaluate multiple solution approaches to provide baseline performance on the benchmark sets. We believe the availability of this dataset and benchmarks will enable realistic, standardized model comparison and foster further advances in phishing detection. The datasets and benchmarks are available on Hugging Face (https://huggingface.co/datasets/phreshphish/phreshphish).

Problem

• Lack of large, high-quality phishing datasets hinders ML progress
• Existing datasets suffer from leakage and unrealistic base rates
• Need realistic benchmarks for standardized phishing detection comparison

Innovation

• Large-scale high-quality phishing website dataset
• Comprehensive benchmark minimizing data leakage
• Realistic base rates and enhanced diversity