Spatio-Temporal Audio Language Modeling for Dynamic Sound Sources

2026-06-12

AI Summary

Existing audio language models struggle to comprehend the spatiotemporal trajectories of dynamic sound sources, while sound source localization models lack semantic reasoning capabilities. To bridge this gap, this work introduces the ST-AudioQA dataset and proposes the ST-Audio Encoder and ST-AudioLM model, which jointly model the semantics of sound events along with their dynamic identity, location, motion, and relational context for the first time. The approach integrates first-order Ambisonic (FOA) audio encoding, temporally resolved representations, trajectory-supervised learning, and a large language model. Experimental results demonstrate that the proposed framework significantly outperforms static spatial and localization-oriented baselines on spatiotemporal audio question answering, achieving a superior balance between semantic understanding and localization accuracy.

Abstract

Sound events are entities with semantic identities, locations, and trajectories, but current audio-language models usually reason about clips as global event content. Conversely, sound event localization models track source directions over time but offer limited semantic coverage for language reasoning. To address this gap, we introduce ST-AudioQA, a spatio-temporal audio QA dataset and benchmark built from first-order ambisonic (FOA) renderings of static and moving sound sources. Each scene provides source identity, activity, direction, distance, and motion metadata, enabling dense trajectory supervision and questions about what is sounding, where it is, how it moves, and how sources relate. We further propose ST-Audio Encoder, a time-resolved FOA audio encoder that learns event semantics together with source trajectories, and ST-AudioLM, which connects the audio tokens from the encoder to an LLM for spatio-temporal audio QA. Experiments show that this representation improves the semantic-localization tradeoff and yields stronger reasoning performance than static spatial and localization-oriented baselines.

Problem

• spatio-temporal audio modeling
• sound event localization
• audio-language reasoning
• dynamic sound sources
• ambisonic audio

Innovation

• spatio-temporal audio modeling
• audio-language model
• sound event localization
• first-order ambisonic (FOA)
• trajectory-aware reasoning