$C^3$ASD: Multi-Level Consistency-Driven Representation Learning

2026-07-03

AI Summary

This work addresses the significant performance degradation of existing active speaker detection methods under real-world disturbances—such as acoustic noise, visual occlusion, or multimodal degradation—and their lack of explicit cross-modal semantic alignment. To overcome these limitations, we propose a multi-level consistency-driven framework that introduces, for the first time, three complementary constraints: embedding-level cross-modal contrastive learning, sequence-level trajectory-aware clustering, and prediction-level knowledge distillation. These mechanisms collectively suppress modality shortcut learning and enable robust cross-modal alignment and fusion. Extensive experiments demonstrate that our approach substantially outperforms state-of-the-art methods across diverse audio, visual, and joint interference scenarios while maintaining superior performance even on clean data.

Abstract

Active Speaker Detection determines whether a visible person in a video is speaking at each moment. While recent audio-visual fusion methods perform well on clean data, they degrade under real-world corruptions such as background noise, occlusion, or simultaneous modality degradation. We attribute this limitation to the absence of explicit consistency constraints that promote robust, semantically aligned representations across modalities. Without such guidance, models tend to learn fragile modality-specific shortcuts that fail under corrupted conditions. We propose $C^3$ASD, a multi-level consistency-driven framework with three complementary constraints: embedding-level inter-modality consistency aligns audio-visual representations during speech; sequence-level intra-modality consistency separates speaking and non-speaking clusters via track-aware contrastive learning; and prediction-level consistency stabilizes fusion through knowledge distillation. Extensive experiments demonstrate significant improvements under diverse audio, visual and joint corruptions, while maintaining competitive performance on clean data.

Problem

• Active Speaker Detection
• audio-visual fusion
• real-world corruptions
• modality degradation
• consistency constraints

Innovation

• multi-level consistency
• audio-visual fusion
• active speaker detection
• contrastive learning
• knowledge distillation