Auditing Generalization in AI-Generated Video Detection: A Six-Control Protocol and the VidAudit Toolkit

2026-06-29

AI Summary

Current benchmarks for detecting AI-generated videos suffer from confounding factors that inflate reported generalization performance. This work proposes the first systematic auditing protocol, integrating six control criteria and introducing Leave-One-Generator-Out (LOGO) evaluation, cross-dataset testing, and real-real consistency checks to disentangle motion forgery signals from dataset-specific artifacts. Applying this framework reveals that several high-AUC detection methods collapse under rigorous evaluation, with AUC dropping to 0.529. To address this, we introduce TemporalSpec—a temporal spectral representation—and Cross-Subspace Feature Fusion (XSFF), enabling WaveRep to maintain a robust 0.996 AUC in out-of-distribution LOGO evaluations. We also release VidAudit, an open-source toolkit integrating 14 detectors with a unified API to support reproducible benchmarking and auditing.

Abstract

AI-generated video detection benchmarks such as GenVidBench and AIGVDBench are the de facto leaderboards, yet most evaluation protocols leave uncontrolled confounds that can inflate reported generalization. As an existence proof, a three-feature clip-length classifier reaches a leave-one-generator-out (LOGO) AUC of 0.998 on GenVidBench under unaudited evaluation, while measuring nothing about motion. A 20-paper survey finds none applying all six standard controls that would catch this, so we combine them into an audited protocol and apply it to six representative feature sources (three published detectors and three repurposed signal sources), re-running it cross-dataset on AIGVDBench. The audit both debunks and certifies: the trivial classifier collapses to near chance (0.529), a CLIP baseline is caught carrying dataset identity, and the 2025 forensic detector WaveRep clears the floor at out-of-distribution LOGO AUC 0.996 with chance-level real-vs-real coherence. At a deployable FPR of 0.1%, multiple high-AUC methods fall to single-digit recall and the leaderboard order changes, so we recommend an audited tuple (AUC, above-floor margin, operating-point recall, and calibration) over a single number. As a white-box positive control, we add TemporalSpec (codec motion vectors); via cross-substrate feature fusion (XSFF), a second substrate adds genuine complementarity that survives the audit. We release VidAudit, to our knowledge the largest unified and audited detector collection for this task, providing 14 detectors behind one plugin API, a leaderboard, and Croissant metadata, available at https://github.com/KurbanIntelligenceLab/vidaudit. Together, the protocol and toolkit move evaluation from leaderboard rank toward whether a result measures what it claims.

Problem

• AI-generated video detection
• generalization
• evaluation protocol
• benchmark confounds
• out-of-distribution

Innovation

• audited evaluation
• generalization audit
• six-control protocol
• cross-substrate feature fusion
• VidAudit toolkit