Auto-AEG: Scalable Data Construction for Open-Vocabulary Audio Event Grounding

2026-07-05

AI Summary

This work addresses the challenge of open-vocabulary audio event localization, which is hindered by the scarcity of large-scale temporally annotated data needed to enable precise temporal interval prediction with large audio-language models. The authors propose a scalable joint framework for data construction and model fine-tuning: first generating cold-start data with exact temporal labels via programmatic synthesis, then leveraging multi-model pseudo-labels to construct reinforcement learning reward signals from real-world audio, thereby enhancing the model’s localization capability under arbitrary natural language queries. This approach achieves, for the first time, fully automatic construction of large-scale open-vocabulary audio event localization data without human annotation and introduces a novel interval-aware reward function. Experiments demonstrate significant performance gains on both DESED SED and AEGBench—a newly introduced, difficulty-stratified benchmark—validating the effectiveness of the proposed framework.

Abstract

Large Audio-Language Models (LALMs) reason fluently about sound yet struggle to localize precisely when events occur, while classical Sound Event Detection attains frame-level precision only over a closed label set. At the intersection of these paradigms lies the task of Open-Vocabulary Audio Event Grounding: predicting all time intervals of a target sound event described by an arbitrary natural language query. While this task is crucial for real-world audio understanding and LALM adaptation, it is bottlenecked by data scarcity. Few large-scale resources provide open-vocabulary onset/offset supervision, and manual temporal annotation is prohibitively expensive. To address this, we introduce Auto-AEG, a scalable pipeline that constructs such supervision by automatic data construction and model fine-tuning. It pairs programmatically synthesized clips, which carry exact ground-truth intervals for supervised cold-start, with multi-model pseudo-labels on real-world audio that supply the reward signal for reinforcement learning. Training with this pipeline yields promising performance gains on both the DESED SED benchmark and AEGBench, an independent difficulty-stratified benchmark we release. Our results show that automatically constructed data, coupled with interval-aware reward function design, is an effective data-side route to expanding the temporal localization capability of LALMs.

Problem

• Open-Vocabulary Audio Event Grounding
• Temporal Localization
• Data Scarcity
• Sound Event Detection
• Audio-Language Models

Innovation

• Auto-AEG
• Open-Vocabulary Audio Event Grounding
• Automatic Data Construction
• Reinforcement Learning
• Temporal Localization