What Do Deepfake Benchmarks Measure? An Audit Using Frozen Self-Supervised Representations

2026-06-24

AI Summary

Current deepfake detection benchmarks may over-rely on general-purpose representational capabilities rather than genuine forensic understanding, leading high-scoring models to fail in real-world scenarios. This work proposes a streamlined auditing approach: training linear probes on frozen, general self-supervised representations and analyzing generator difficulty via Fréchet distance to systematically assess what underlying capabilities these benchmarks actually measure. Experiments across image, video, and audio modalities reveal that linear probes achieve performance comparable to specialized detectors, suggesting that most existing benchmarks have already been “solved” by generic representations. This finding calls into question the true forensic competence of state-of-the-art models that attain high scores on current benchmarks.

Abstract

As deepfake generators approach perceptual indistinguishability, reliable detection becomes critical. Yet, detectors that score well on benchmarks routinely fail in the wild. A concerning feedback loop has emerged: benchmarks drive increasingly complex, engineered detectors, yet if those benchmarks do not reflect real-world deepfakes, this complexity may be solving the wrong problem entirely. This raises a prior question: what are these benchmarks actually measuring? We conduct an audit of video, image, and audio deepfake benchmarks using a deliberately simple diagnostic. If a linear probe on frozen, general-purpose self-supervised representations can approximate the performance of a bespoke detector, the benchmark is largely rewarding general modality understanding rather than forensic understanding. This has two implications: the benchmark may not reflect realistic threat models, and it raises the question of whether the bespoke detectors the probe approaches are truly learning forensic understanding. We observe, across three modalities, linear probes on general-purpose self-supervised representations closely approach the performance of bespoke detectors. We further show that generator-level difficulty is partly explained by Frechet geometry in the same representation space. Together, these results support a benchmark-audit view of deepfake detection: before high scores are read as evidence of forensic understanding, it is worth asking how much of the benchmark is already solved by general-purpose representations.

Problem

• deepfake detection
• benchmarks
• forensic understanding
• self-supervised representations
• generalization

Innovation

• deepfake detection
• self-supervised representations
• benchmark audit
• linear probe
• Fréchet geometry