EgoSound: Benchmarking Sound Understanding in Egocentric Videos

2026-02-15

AI Summary

This work addresses the lack of systematic evaluation of auditory understanding capabilities in existing vision-language models when applied to egocentric videos. To this end, we introduce the first benchmark specifically designed for auditory comprehension in this setting, proposing a comprehensive evaluation framework encompassing seven task categories—including spatial localization and causal reasoning. Leveraging a multi-stage pipeline of automatic generation and human validation, we integrate data from Ego4D and EgoBlind to construct a high-quality evaluation set comprising 900 videos and 7,315 question-answer pairs. Experiments on nine state-of-the-art multimodal large language models reveal that while current models exhibit preliminary auditory reasoning abilities, they remain significantly limited in fine-grained spatial and causal understanding.

Technology Category

• Computer Vision: Language and Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Multimodal Large Language Models (MLLMs) have recently achieved remarkable progress in vision-language understanding. Yet, human perception is inherently multisensory, integrating sight, sound, and motion to reason about the world. Among these modalities, sound provides indispensable cues about spatial layout, off-screen events, and causal interactions, particularly in egocentric settings where auditory and visual signals are tightly coupled. To this end, we introduce EgoSound, the first benchmark designed to systematically evaluate egocentric sound understanding in MLLMs. EgoSound unifies data from Ego4D and EgoBlind, encompassing both sighted and sound-dependent experiences. It defines a seven-task taxonomy spanning intrinsic sound perception, spatial localization, causal inference, and cross-modal reasoning. Constructed through a multi-stage auto-generative pipeline, EgoSound contains 7315 validated QA pairs across 900 videos. Comprehensive experiments on nine state-of-the-art MLLMs reveal that current models exhibit emerging auditory reasoning abilities but remain limited in fine-grained spatial and causal understanding. EgoSound establishes a challenging foundation for advancing multisensory egocentric intelligence, bridging the gap between seeing and truly hearing the world.

Problem

• egocentric sound understanding
• multimodal large language models
• auditory reasoning
• cross-modal reasoning
• sound perception

Innovation

• EgoSound
• egocentric sound understanding
• multimodal large language models
• audio-visual reasoning
• benchmark