SCoPE: Training-Free Audio-Visual Event Perception via Sparse Cross-Modal Prior Exchange

2026-08-08

AI Summary

This work addresses the false co-activation (FCA) problem in training-free audio-visual event perception, which arises from shared evidential support across labels. To mitigate this issue, the authors propose a training-free cross-modal competition framework that introduces, for the first time, a sparse cross-modal prior exchange mechanism. This enables mutual guidance between audio and visual modalities and precisely eliminates FCA through competitive querying of label pairs over shared evidence, with theoretical elimination conditions derived for the two-label case. Leveraging frozen CLIP and CLAP models along with text-encoded event names, the method achieves significant improvements on the LLP dataset—gaining 7.45 and 5.04 points in Type@seg and Event@seg metrics, respectively—and demonstrates strong generalization performance on OV-AVEBench and VGGSound-AVEL100k.

Abstract

Audio-visual event perception (AVEP) determines which events occur in a video, when they occur, and whether they are audible, visible, or both. Training-free methods query new event vocabularies by matching frozen audio and visual features with text-encoded event names. However, related labels share evidence. An incorrect label can then score at least as high as a correct one. We call this a false co-activation (FCA). No scalar cutoff can reject the incorrect label while keeping every correct one. Class-specific thresholds may prevent that label from becoming a final prediction, but the FCA remains in the underlying score vector. We introduce SCoPE, a training-free framework in which all queried labels compete for shared evidence and each modality guides event selection in the other. We derive an exact condition for when this competition removes an FCA in a two-label fit. With identical frozen CLIP+CLAP backbones on LLP, SCoPE improves Type@seg by 7.45 points and Event@seg by 5.04 points compared with the reported AV$^2$A values. The same fixed configuration transfers unchanged to OV-AVEBench and VGGSound-AVEL100k.

Problem

• audio-visual event perception
• training-free
• false co-activation
• cross-modal
• event recognition

Innovation

• training-free
• audio-visual event perception
• false co-activation
• cross-modal competition
• frozen features