SceneBind: Binding What and Where Across Vision, Audio and Language

2026-07-16

AI Summary

Existing multimodal encoders excel at instance-level semantic understanding (“what is present”) but lack explicit modeling of spatial structure (“where it is”). This work proposes SceneBind, the first framework to jointly model semantic and spatial information across vision, audio, and language modalities. By introducing lightweight, object-centric semantic-spatial slots, SceneBind explicitly represents object-level semantics, 3D spatial attributes, and their associated uncertainties—while preserving the capabilities of large-scale pretrained encoders. The method further incorporates a SceneBind Matching mechanism to enable cross-modal scene retrieval and object localization. Requiring only a small number of additional spatial tokens, SceneBind achieves state-of-the-art performance on scene and spatial retrieval benchmarks and demonstrates exceptional zero-shot transfer capabilities on downstream tasks such as audio-visual localization.

Abstract

We present SceneBind, an omni-modal representation of realistic scenes with joint semantic and 3D spatial understanding across vision, audio and language. Existing omni-modal encoders excel at instance-level semantics (i.e., what is present), but often lack explicit spatial structure (i.e., where it is). SceneBind addresses this gap by representing each scene as a semantic-spatial entity, combining a global semantic embedding with object-centric semantic-spatial slots. This representation explicitly captures object-level semantics, spatial attributes, and uncertainty. We further propose SceneBind Matching, a semantic-spatial matching scheme that integrates global scene similarity with object alignment, supporting cross-modal scene retrieval and object grounding. To train and evaluate SceneBind, we curate a novel real-world binaural audio-visual dataset with structured semantic and spatial annotations, and propose a training protocol for aligning semantic and spatial signals across modalities. SceneBind is compatible with large-scale pretrained semantic encoders, adds lightweight spatial modeling with only a few additional tokens. It achieves state-of-the-art scene and spatial retrieval while enabling strong zero-shot transfer to downstream tasks such as audio-visual localization.

Problem

• omni-modal representation
• semantic understanding
• spatial understanding
• scene representation
• cross-modal alignment

Innovation

• omni-modal representation
• semantic-spatial binding
• object-centric slots
• cross-modal retrieval
• zero-shot transfer