AV-SyncBench: Decoupled Benchmarking of Temporal and Semantic Audio-Visual Synchronization

2026-07-01

AI Summary

This work addresses the limitations of existing audio-visual synchronization evaluation methods, which struggle to disentangle temporal alignment from semantic consistency and suffer from coupling biases in data construction. We propose the first structured and scalable benchmark framework that enables independent assessment of temporal synchronization and semantic correspondence. Through a hybrid pipeline combining automated filtering and human verification, we construct a large-scale dataset comprising 3,269 videos and 38,390 samples across three audio categories—speech, music, and environmental sounds—and ten diverse scenarios. The dataset ensures authentic on-screen sound sources and supports both multimodal alignment analysis and downstream task evaluation. Using this benchmark, we systematically evaluate five representative models. Both code and data are publicly released.

Abstract

Audio-visual feature extraction is a fundamental component of multimodal understanding and generation tasks. However, existing evaluation protocols for feature extraction models exhibit dimensional bias, typically focusing on either semantic matching or temporal offset detection. Moreover, their data construction remains coupled, preventing independent assessment of temporal and semantic consistency. We propose AV-SyncBench, the first benchmark to fully separate temporal and semantic evaluation for audio-visual synchronization. Built from in-the-wild videos, it spans Voice, Music, and Sound across 10 scenarios and 5 challenge tasks. Data are automatically filtered and manually verified to ensure on-screen sound sources. The benchmark contains 3,269 videos and 38,390 samples, and we evaluate five representative models to quantify feature quality for alignment and downstream tasks. The code and dataset are available at: https://fgt7t6g.github.io/AV-SyncBench.

Problem

• audio-visual synchronization
• temporal alignment
• semantic consistency
• evaluation benchmark
• multimodal understanding

Innovation

• audio-visual synchronization
• decoupled benchmarking
• temporal alignment
• semantic consistency
• multimodal evaluation