Environmental Sound Deepfake Detection Using Deep-Learning Framework

2026-04-21

AI Summary

This study addresses the detection of deepfakes in environmental audio by systematically treating sound scene and sound event forgery as two distinct tasks. The authors propose a three-stage fine-tuning strategy based on the WavLM pretrained model, integrating multiple spectrogram representations and network architectures, which substantially outperforms training from scratch. Experimental results demonstrate that the proposed method achieves an accuracy of 0.98, F1 score of 0.95, and AUC of 0.99 on the EnvSDD test set, and an accuracy of 0.88, F1 score of 0.77, and AUC of 0.92 on the ESDD-Challenge-TestSet. These findings validate the effectiveness and generalization capability of fine-tuning pretrained models for audio deepfake detection.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Computer Vision: Diffusion Models for Vision

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

In this paper, we propose a deep-learning framework for environmental sound deepfake detection (ESDD) -- the task of identifying whether the sound scene and sound event in an input audio recording is fake or not. To this end, we conducted extensive experiments to explore how individual spectrograms, a wide range of network architectures and pre-trained models, ensemble of spectrograms or network architectures affect the ESDD task performance. The experimental results on the benchmark datasets of EnvSDD and ESDD-Challenge-TestSet indicate that detecting deepfake audio of sound scene and detecting deepfake audio of sound event should be considered as individual tasks. We also indicate that the approach of finetuning a pre-trained model is more effective compared with training a model from scratch for the ESDD task. Eventually, our best model, which was finetuned from the pre-trained WavLM model with the proposed three-stage training strategy, achieve the Accuracy of 0.98, F1 Score of 0.95, AuC of 0.99 on EnvSDD Test subset and the Accuracy of 0.88, F1 Score of 0.77, and AuC of 0.92 on ESDD-Challenge-TestSet dataset.

Problem

• environmental sound
• deepfake detection
• sound scene
• sound event
• audio forgery

Innovation

• Environmental Sound Deepfake Detection
• Pre-trained Model Fine-tuning
• Three-stage Training Strategy
• WavLM
• Sound Scene and Event Disentanglement