VendorBench-100: A Unified Cross-Paradigm Benchmark for Deepfake Image Detection

2026-07-07

AI Summary

This work addresses the lack of a unified evaluation protocol that hinders direct comparison among deepfake detection approaches across three dominant paradigms: commercial APIs, zero-shot vision-language models, and open-source detectors. To bridge this gap, the authors introduce VendorBench-100, a benchmark comprising a standardized corpus of 100 adversarial images, evaluated under consistent output formatting and a dual-metric strategy prioritizing the Matthews Correlation Coefficient (MCC) with ROC-AUC as a secondary measure. For the first time, 36 representative models are systematically assessed within a common framework. The results reveal that commercial APIs generally achieve the best performance, while certain open-source models rival top-tier vision-language models. Notably, high ROC-AUC scores do not necessarily correspond to high MCC values, underscoring the critical influence of metric selection on evaluation outcomes.

Abstract

Deepfake image detection is currently served by three fundamentally different paradigms: commercial APIs, zero-shot vision-language models (LLMs), and open-source detectors. Despite their widespread use, these paradigms are rarely evaluated under a common protocol, making direct comparison difficult. We introduce VendorBench-100, a cross-paradigm benchmark that evaluates 36 representative models using a single adversarial 100-image corpus, a unified output schema, and a common evaluation framework. To ensure reliable assessment under the corpus's intentional class imbalance, models are ranked primarily by the Matthews correlation coefficient (MCC), with ROC-AUC reported as a threshold-independent measure of ranking ability. Rather than maximizing dataset size, VendorBench-100 emphasizes challenging real-world scenarios through a curated taxonomy of eight edge-case families, including face swaps, text-to-video stills, AI photo edits, avatar compositing, opaque-provenance images, and compressed research frames. Our evaluation shows that commercial APIs achieve the strongest median performance, followed by vision LLMs and open-source detectors. However, individual open-source models remain competitive with the best vision LLMs. More importantly, we identify a consistent divergence between ranking ability (ROC-AUC) and operating-point quality (MCC), demonstrating that strong score discrimination does not necessarily produce reliable default-threshold decisions. This metric disagreement, rather than any single leaderboard ranking, is the central finding of the benchmark. We release the complete evaluation framework and benchmark results to support reproducible future research. The source code and data are available at: https://github.com/sharayu-20/vendorbench-100

Problem

• deepfake detection
• cross-paradigm benchmark
• evaluation protocol
• model comparison
• adversarial images

Innovation

• cross-paradigm benchmark
• deepfake detection
• Matthews correlation coefficient
• adversarial image corpus
• vision-language models