BAT: Learning to Reason about Spatial Sounds with Large Language Models

2024-02-02

International Conference on Machine Learning

AI Summary

To address the scarcity of real-world spatial audio data and the lack of spatial-semantic reasoning capabilities in existing models, this paper proposes the first framework integrating binaural acoustic perception with language-based reasoning. Methodologically, we design Spatial-AST—a spatial audio encoder—and introduce SpatialSoundQA, the first in-the-wild, binaural spatial audio question-answering dataset. Furthermore, we pioneer the use of a large language model (LLaMA-2 7B) for spatial causal and multi-step relational reasoning. Contributions include: (1) a paradigm shift from conventional sound event localization and detection (SELD) toward spatial understanding and semantic reasoning; (2) Spatial-AST achieving state-of-the-art performance in sound event detection, localization, and distance estimation; and (3) our BAT model significantly outperforming baselines on both spatial perception and multi-step reasoning tasks.

Technology Category

• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Spatial sound reasoning is a fundamental human skill, enabling us to navigate and interpret our surroundings based on sound. In this paper we present BAT, which combines the spatial sound perception ability of a binaural acoustic scene analysis model with the natural language reasoning capabilities of a large language model (LLM) to replicate this innate ability. To address the lack of existing datasets of in-the-wild spatial sounds, we synthesized a binaural audio dataset using AudioSet and SoundSpaces 2.0. Next, we developed SpatialSoundQA, a spatial sound-based question-answering dataset, offering a range of QA tasks that train BAT in various aspects of spatial sound perception and reasoning. The acoustic front end encoder of BAT is a novel spatial audio encoder named Spatial Audio Spectrogram Transformer, or Spatial-AST, which by itself achieves strong performance across sound event detection, spatial localization, and distance estimation. By integrating Spatial-AST with LLaMA-2 7B model, BAT transcends standard Sound Event Localization and Detection (SELD) tasks, enabling the model to reason about the relationships between the sounds in its environment. Our experiments demonstrate BAT's superior performance on both spatial sound perception and reasoning, showcasing the immense potential of LLMs in navigating and interpreting complex spatial audio environments.

Problem

• Replicating human spatial sound reasoning using LLMs and binaural models
• Addressing lack of datasets with synthesized binaural audio data
• Enhancing SELD tasks with integrated audio-language reasoning

Innovation

• Combines binaural audio analysis with LLM reasoning
• Uses synthesized dataset from AudioSet and SoundSpaces
• Introduces Spatial-AST encoder for audio processing