🤖 AI Summary
This work addresses the challenge of cross-modal noise propagation caused by unidirectional or asynchronous manipulations in audio-visual deepfakes. To this end, we propose IaMSB, a novel framework that, for the first time, introduces the Schrödinger bridge into forgery localization. Our method employs a lightweight coarse bridge to screen candidate intervals and estimate cross-modal consistency, followed by a refined bridge that performs asymmetric step optimization and bottlenecked cross-modal interaction to achieve high-precision interval-level temporal localization. IaMSB unifies cross-modal consistency modeling, informative segment selection, and bridge-step scheduling within a single architecture. Extensive experiments demonstrate significant performance gains under strict IoU thresholds across multiple benchmarks, with AP@0.95 improvements of 3%–10%, particularly excelling in scenarios involving unidirectional forgeries.
📝 Abstract
Audio-visual deepfake localization demands interval-level outputs that serve as temporal evidence. Despite recent progress, symmetric fusion under single-sided or asynchronous forgeries propagates cross-modal noise, degrading high-precision localization. We present IaMSB, an inconsistency-aware multimodal Schrödinger Bridge (SB) that jointly estimates cross-modal consistency and performs interval-level localization. Unlike diffusion models, SB minimizes path-distribution discrepancy and yields consistency scores without explicit noise injection or denoising. With the Schrödinger Bridge (SB), IaMSB unifies consistency estimation, cross-modal information selection, and bridge-step scheduling in one framework. Specifically, a lightweight coarse bridge first proposes candidate intervals and estimates cross-modal consistency; these statistics select cross-modal witness signals and allocate bridge steps asymmetrically across modalities. A refinement bridge then performs step-tuned fusion and outputs refined, time-aligned intervals. IaMSB anticipates single-sided and asynchronous forgeries and, using bottlenecked cross-modal interaction with step allocation, suppresses noise transfer, avoids unnecessary iterations. Across benchmarks, IaMSB stabilizes strict-IoU boundary precision, raising AP@0.95 by 3%~10%, and yields improved high-precision localization, particularly for single-sided forgeries.