TimeAudio: Bridging Temporal Gaps in Large Audio-Language Models

2025-11-14

AI Summary

Large audio-language models face significant bottlenecks in temporal localization (e.g., temporal audio grounding) and long-audio understanding, stemming from coarse timestamp representations, architectures lacking explicit temporal modeling capabilities, and scarcity of high-quality temporally annotated data. To address these challenges, we propose Time-ALM: a novel framework featuring a temporal token enhancement module to improve semantic alignment of timestamps; absolute time encoding to enable continuous-time awareness; and a paragraph-level token merging mechanism to support long-audio modeling. Leveraging end-to-end multi-task training and cross-modal alignment, we construct the first dedicated dataset and benchmark for fine-grained temporal understanding. Experiments demonstrate that Time-ALM substantially outperforms state-of-the-art methods on dense captioning, temporal localization, and timeline summarization—achieving, for the first time, high-accuracy, long-horizon, and interpretable audio temporal understanding.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Planning, Routing, and Scheduling: Temporal Planning

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Recent Large Audio-Language Models (LALMs) exhibit impressive capabilities in understanding audio content for conversational QA tasks. However, these models struggle to accurately understand timestamps for temporal localization (e.g., Temporal Audio Grounding) and are restricted to short audio perception, leading to constrained capabilities on fine-grained tasks. We identify three key aspects that limit their temporal localization and long audio understanding: (i) timestamp representation, (ii) architecture, and (iii) data. To address this, we introduce TimeAudio, a novel method that empowers LALMs to connect their understanding of audio content with precise temporal perception. Specifically, we incorporate unique temporal markers to improve time-sensitive reasoning and apply an absolute time-aware encoding that explicitly grounds the acoustic features with absolute time information. Moreover, to achieve end-to-end long audio understanding, we introduce a segment-level token merging module to substantially reduce audio token redundancy and enhance the efficiency of information extraction. Due to the lack of suitable datasets and evaluation metrics, we consolidate existing audio datasets into a new dataset focused on temporal tasks and establish a series of metrics to evaluate the fine-grained performance. Evaluations show strong performance across a variety of fine-grained tasks, such as dense captioning, temporal grounding, and timeline speech summarization, demonstrating TimeAudio's robust temporal localization and reasoning capabilities.

Problem

• Improving timestamp understanding for temporal audio localization in models
• Enhancing long audio perception beyond short audio limitations
• Addressing timestamp representation, architecture, and data limitations in LALMs

Innovation

• Incorporates unique temporal markers for time-sensitive reasoning
• Applies absolute time-aware encoding to ground acoustic features
• Introduces segment-level token merging to reduce audio redundancy