UniAV: Unified Audio-Visual Perception for Multi-Task Video Event Localization

2024-04-04

IEEE Transactions on Pattern Analysis and Machine Intelligence

AI Summary

Existing video event localization methods—temporal action localization (TAL), sound event detection (SED), and audio-visual event localization (AVEL)—operate in isolation, hindering holistic video understanding. Moreover, their divergent feature preferences and imbalanced dataset distributions limit the effectiveness of naive multi-task learning. To address this, we propose UniAVL, a unified audio-visual perception framework featuring a shared multi-scale audio-visual encoder and task-specific expert networks, coupled with a semantics-aligned language-prompted classifier enabling open-set novel-class localization. By jointly optimizing cross-modal feature sharing and prompt-based adaptation, UniAVL achieves synergistic learning across all three tasks. On ActivityNet 1.3, DESED, and UnAV-100, UniAVL consistently outperforms both single-task and vanilla multi-task baselines, matching or exceeding state-of-the-art performance per task—marking the first unified modeling and joint improvement of TAL, SED, and AVEL.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Video event localization tasks include temporal action localization (TAL), sound event detection (SED) and audio-visual event localization (AVEL). Existing methods tend to over-specialize on individual tasks, neglecting the equal importance of these different events for a complete understanding of video content. In this work, we aim to develop a unified framework to solve TAL, SED and AVEL tasks together to facilitate holistic video understanding. However, it is challenging since different tasks emphasize distinct event characteristics and there are substantial disparities in existing task-specific datasets (size/domain/duration). It leads to unsatisfactory results when applying a naive multi-task strategy. To tackle the problem, we introduce UniAV, a Unified Audio-Visual perception network to effectively learn and share mutually beneficial knowledge across tasks and modalities. Concretely, we propose a unified audio-visual encoder to derive generic representations from multiple temporal scales for videos from all tasks. Meanwhile, task-specific experts are designed to capture the unique knowledge specific to each task. Besides, instead of using separate prediction heads, we develop a novel unified language-aware classifier by utilizing semantic-aligned task prompts, enabling our model to flexibly localize various instances across tasks with an impressive open-set ability to localize novel categories. Extensive experiments demonstrate that UniAV, with its unified architecture, significantly outperforms both single-task models and the naive multi-task baseline across all three tasks. It achieves superior or on-par performances compared to the state-of-the-art task-specific methods on ActivityNet 1.3, DESED and UnAV-100 benchmarks.

Problem

• Unified framework for multi-task video event localization challenges
• Overcoming task-specific dataset disparities and modality differences
• Developing shared representations while preserving task-specific knowledge

Innovation

• Unified audio-visual encoder for multi-scale representations
• Task-specific experts capturing unique event characteristics
• Language-aware classifier with semantic-aligned prompts