Problem
Research questions and friction points this paper is trying to address.
Audio-Visual Temporal Forgery Localization
frequency-aware modeling
multimodal fusion
high-frequency noise
robustness
UniSkip-Mamba: A Frequency-Aware State Space Model for Audio-Visual Temporal Forgery Localization
🤖 AI Summary
Existing audio-visual temporal forgery localization methods treat all frequency bands uniformly, rendering them susceptible to high-frequency noise and lacking robustness. This work reveals for the first time that discriminative forgery cues are predominantly concentrated in the low-to-mid frequency range (0–0.15 Hz) and introduces a frequency-aware state space model. The proposed architecture integrates multimodal sequence fusion within a unified framework and incorporates a Skip-Scan Mamba module enhanced with a Group-Scan-Merge mechanism to enforce frequency-domain regularization. This design preserves representational completeness while significantly improving noise resilience. Evaluated on LAV-DF, the method achieves 63.4% AP@0.95 (+9.8%), and on AV-Deepfake1M, it attains 63.58% mAP (+14.32%), with a sixfold increase in inference speed.
📝 Abstract
With the proliferation of AI-generated content, sophisticated multimedia manipulation has raised critical concerns about malicious applications such as opinion manipulation and evidence fabrication, making Audio-Visual Temporal Forgery Localization (AV-TFL) an urgent research frontier. Existing TFL methods have progressed along two main paradigms: Transformer-based temporal modeling and channel-wise multimodal fusion. While these approaches capture temporal dependencies and cross-modal correlations, they process all frequency components indiscriminately, leading to overfitting on high-frequency noise and limited robustness under real-world data degradation. Through systematic frequency domain analysis, we find that forgery-discriminative patterns concentrate in the low/mid-frequency range (normalized frequency 0-0.15), while high-frequency components primarily introduce noise, removing them even improves detection performance by +1.4%. Based on this phenomenon, we propose UniSkip-Mamba, a frequency-aware State Space Model framework that incorporates Unified Multimodal Sequence Fusion to preserve cross-modal phase relationships, and Skip-Scanning Mamba Blocks that implement frequency-aware regularization through a novel Group-Scan-Merge mechanism, naturally biasing learning toward discriminative low/mid-frequency patterns (0-0.15) while maintaining representational completeness. We achieve state-of-the-art (SOTA) performance: 63.4% AP@0.95 on LAV-DF (+9.8% improvement) and 63.58% mAP on AV-Deepfake1M (+14.32% improvement), with 6x faster inference. Our frequency-domain analysis provides theoretical justification from a signal processing perspective for why skip-scanning inherently improves both accuracy and robustness.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Frequency-Aware
State Space Model
Multimodal Fusion
Temporal Forgery Localization
Skip-Scanning
🔎 Similar Papers
No similar papers found.
