Semantic Contextualization of Face Forgery: A New Definition, Dataset, and Detection Method

2024-05-14

arXiv.org

AI Summary

This study addresses the fundamental question of “which digital manipulations cause a face image to be classified as forged,” redefining face forgery semantically for the first time as manipulations of semantic attributes exceeding human discriminability thresholds. Method: We construct the first forgery dataset annotated with hierarchical semantic labels—including facial parts, texture, illumination, and identity—and propose a semantic-aware detection framework integrating semantic graph modeling, hierarchical label learning, multi-task collaborative optimization, and semantic-guided feature attention. Contributions/Results: Experiments reveal that our dataset exposes critical generalization failures of mainstream detectors. When used for training, it significantly improves cross-attack robustness. Our method achieves substantial performance gains on both binary and multi-class forgery detection tasks, empirically validating that explicit semantic modeling delivers intrinsic benefits for deepfake detection.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Machine Learning: Adversarial Learning & Robustness

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

In recent years, deep learning has greatly streamlined the process of manipulating photographic face images. Aware of the potential dangers, researchers have developed various tools to spot these counterfeits. Yet, none asks the fundamental question: extit{What digital manipulations make a real photographic face image fake, while others do not?} In this paper, we put face forgery in a semantic context and define that extit{computational methods that alter semantic face attributes to exceed human discrimination thresholds are sources of face forgery}. Following our definition, we construct a large face forgery image dataset, where each image is associated with a set of labels organized in a hierarchical graph. Our dataset enables two new testing protocols to probe the generalizability of face forgery detectors. Moreover, we propose a semantics-oriented face forgery detection method that captures label relations and prioritizes the primary task (ie, real or fake face detection). We show that the proposed dataset successfully exposes the weaknesses of current detectors as the test set and consistently improves their generalizability as the training set. Additionally, we demonstrate the superiority of our semantics-oriented method over traditional binary and multi-class classification-based detectors.

Problem

• Defining semantic criteria for face forgery detection
• Creating a hierarchical dataset for forgery testing
• Developing a semantics-oriented forgery detection method

Innovation

• Defines face forgery via semantic attribute alterations
• Creates hierarchical labeled dataset for forgery detection
• Proposes semantics-oriented detection method improving generalizability