Sequential Contrastive Audio-Visual Learning

2024-07-08

IEEE International Conference on Acoustics, Speech, and Signal Processing

AI Summary

Traditional audio-video contrastive learning relies on temporal aggregation of representations, discarding fine-grained sequential structure and thereby limiting cross-modal alignment accuracy. To address this, we propose the first framework that abandons temporal aggregation entirely and performs contrastive learning directly in the raw sequence space. Our method introduces an end-to-end audio-video Transformer encoder, a DTW-enhanced multi-dimensional temporal distance metric, and a novel sequence-level contrastive loss. The resulting model supports flexible retrieval metrics and efficient hybrid retrieval. Evaluated on VGGSound and Music datasets, our approach achieves up to a 3.5× improvement in audio-video retrieval recall, while using fewer parameters and requiring significantly less training data. It substantially enhances temporal-aware cross-modal representation learning, enabling more precise alignment of fine-grained audio and video dynamics.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Contrastive learning has emerged as a powerful technique in audio-visual representation learning, leveraging the natural co-occurrence of audio and visual modalities in webscale video datasets. However, conventional contrastive audio-visual learning (CAV) methodologies often rely on aggregated representations derived through temporal aggregation, neglecting the intrinsic sequential nature of the data. This oversight raises concerns regarding the ability of standard approaches to capture and utilize fine-grained information within sequences. In response to this limitation, we propose sequential contrastive audiovisual learning (SCAV), which contrasts examples based on their non-aggregated representation space using multidimensional sequential distances. Audio-visual retrieval experiments with the VGGSound and Music datasets demonstrate the effectiveness of SCAV, with up to 3.5x relative improvements in recall against traditional aggregation-based contrastive learning and other previously proposed methods, which utilize more parameters and data. We also show that models trained with SCAV exhibit a significant degree of flexibility regarding the metric employed for retrieval, allowing us to use a hybrid retrieval approach that is both effective and efficient.

Problem

• Captures fine-grained sequential audio-visual data details
• Improves retrieval performance over traditional aggregation methods
• Enables flexible and efficient hybrid retrieval approaches

Innovation

• Sequential contrastive learning for audio-visual data
• Uses multidimensional sequential distances for representation
• Hybrid retrieval approach enhances efficiency and effectiveness