Speech DF Arena: A Leaderboard for Speech DeepFake Detection Models

2025-09-02

AI Summary

Current deepfake speech detection research lacks standardized, comprehensive benchmarks, hindering fair performance comparison and reproducibility. To address this, we introduce Speech DF Arena—the first holistic benchmark platform for deepfake speech detection. It integrates 14 diverse datasets and attack scenarios, defines a unified evaluation protocol and metrics (e.g., Equal Error Rate, EER), and supports cross-domain generalization assessment. We open-source a full-stack toolkit on GitHub and deploy a live leaderboard on Hugging Face to enable continuous ranking and community collaboration. Our systematic evaluation of 12 open-source and 3 proprietary detectors reveals a significant robustness degradation under cross-domain conditions—manifested by substantial EER increases across all models. Speech DF Arena advances the field toward transparency, reproducibility, and rigorous comparability, providing a foundational infrastructure for algorithmic development and standardization efforts.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Large Multimodal Models (LMMs)
• Application Domains: Misinformation & Fake News

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

Parallel to the development of advanced deepfake audio generation, audio deepfake detection has also seen significant progress. However, a standardized and comprehensive benchmark is still missing. To address this, we introduce Speech DeepFake (DF) Arena, the first comprehensive benchmark for audio deepfake detection. Speech DF Arena provides a toolkit to uniformly evaluate detection systems, currently across 14 diverse datasets and attack scenarios, standardized evaluation metrics and protocols for reproducibility and transparency. It also includes a leaderboard to compare and rank the systems to help researchers and developers enhance their reliability and robustness. We include 14 evaluation sets, 12 state-of-the-art open-source and 3 proprietary detection systems. Our study presents many systems exhibiting high EER in out-of-domain scenarios, highlighting the need for extensive cross-domain evaluation. The leaderboard is hosted on Huggingface1 and a toolkit for reproducing results across the listed datasets is available on GitHub.

Problem

• Lack of standardized benchmark for audio deepfake detection evaluation
• Need comprehensive cross-domain testing for detection model reliability
• Requirement for reproducible evaluation protocols across diverse datasets

Innovation

• Comprehensive benchmark for audio deepfake detection
• Toolkit for uniform evaluation across datasets
• Standardized metrics and protocols for reproducibility