PartialEdit: Identifying Partial Deepfakes in the Era of Neural Speech Editing

2025-06-03

AI Summary

This work addresses the challenge of detecting locally manipulated deepfake speech—audio in which only specific segments are altered via neural speech editing techniques. We introduce PartialEdit, the first dedicated benchmark dataset supporting both forgery detection and tampering localization. We formally define “partially edited deepfake speech” as a novel threat model and propose a unified benchmark framework jointly optimizing detection and localization. Crucially, we identify and model learnable forensic traces introduced by neural audio codecs, and design a feature extraction module integrating contrastive learning with spectral residual analysis. On PartialEdit, state-of-the-art detection models suffer a sharp accuracy drop to 52%, whereas our method achieves 98.3% detection accuracy and 86.7% localization IoU—demonstrating substantial improvements in robustness and practical applicability for real-world partial-editing scenarios.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Responsible Web: Data and user privacy-enhancing technologies for the Web
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Neural speech editing enables seamless partial edits to speech utterances, allowing modifications to selected content while preserving the rest of the audio unchanged. This useful technique, however, also poses new risks of deepfakes. To encourage research on detecting such partially edited deepfake speech, we introduce PartialEdit, a deepfake speech dataset curated using advanced neural editing techniques. We explore both detection and localization tasks on PartialEdit. Our experiments reveal that models trained on the existing PartialSpoof dataset fail to detect partially edited speech generated by neural speech editing models. As recent speech editing models almost all involve neural audio codecs, we also provide insights into the artifacts the model learned on detecting these deepfakes. Further information about the PartialEdit dataset and audio samples can be found on the project page: https://yzyouzhang.com/PartialEdit/index.html.

Problem

• Detecting partially edited deepfake speech segments
• Localizing edits in neural speech editing deepfakes
• Addressing artifacts from neural audio codecs in detection

Innovation

• Introduces PartialEdit dataset for deepfake speech
• Explores detection and localization of neural edits
• Analyzes artifacts from neural audio codecs