AudioMarathon: A Comprehensive Benchmark for Long-Context Audio Understanding and Efficiency in Audio LLMs

2025-10-08

AI Summary

Large Audio-Language Models (LALMs) face dual challenges in long-audio processing: quadratic attention complexity (O(N²)) and weak temporal dependency modeling; existing benchmarks predominantly target short audio, lacking evaluation capability for long-context understanding. This work introduces the first comprehensive benchmark for long-duration audio understanding, spanning speech, sound, and music domains, supporting input lengths of 2,250–7,500 audio tokens. It systematically establishes three novel evaluation dimensions: long-sequence comprehension, cross-domain generalization, and multi-hop reasoning. Leveraging optimization techniques—including token pruning and KV cache eviction—we conduct joint performance-efficiency evaluation across state-of-the-art LALMs. Results expose critical bottlenecks: sharp accuracy degradation and severe memory overhead under long-sequence inference. The benchmark provides a reproducible evaluation framework and concrete architectural guidance for developing efficient, scalable audio foundation models.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Processing long-form audio is a major challenge for Large Audio Language models (LALMs). These models struggle with the quadratic cost of attention ($O(N^2)$) and with modeling long-range temporal dependencies. Existing audio benchmarks are built mostly from short clips and do not evaluate models in realistic long context settings. To address this gap, we introduce AudioMarathon, a benchmark designed to evaluate both understanding and inference efficiency on long-form audio. AudioMarathon provides a diverse set of tasks built upon three pillars: long-context audio inputs with durations ranging from 90.0 to 300.0 seconds, which correspond to encoded sequences of 2,250 to 7,500 audio tokens, respectively, full domain coverage across speech, sound, and music, and complex reasoning that requires multi-hop inference. We evaluate state-of-the-art LALMs and observe clear performance drops as audio length grows. We also study acceleration techniques and analyze the trade-offs of token pruning and KV cache eviction. The results show large gaps across current LALMs and highlight the need for better temporal reasoning and memory-efficient architectures. We believe AudioMarathon will drive the audio and multimodal research community to develop more advanced audio understanding models capable of solving complex audio tasks.

Problem

• Evaluating long-context audio understanding and efficiency in LALMs
• Addressing quadratic attention cost and long-range dependencies in audio
• Benchmarking models on multi-hop reasoning across diverse audio domains

Innovation

• Introduces AudioMarathon benchmark for long-context audio evaluation
• Evaluates models on multi-hop reasoning across speech, sound, music
• Analyzes token pruning and KV cache eviction acceleration techniques