PolyBench: A Benchmark for Compositional Reasoning in Polyphonic Audio

2026-03-05

AI Summary

This work addresses the absence of effective evaluation benchmarks for compositional reasoning about concurrent sound events and their relationships in polyphonic audio. To this end, we present PolyBench, the first systematically constructed benchmark comprising five subtasks—counting, classification, detection, concurrency judgment, and duration estimation—to comprehensively assess models’ compositional reasoning capabilities. Experiments on prominent large audio language models reveal a significant performance drop in polyphonic scenarios, underscoring the challenge and validity of PolyBench while exposing critical limitations of current models in handling complex audio understanding tasks involving multiple simultaneous sounds.

Technology Category

• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Large Audio Language Models (LALMs) are increasingly capable of reasoning over audio. However, existing benchmarks provide limited coverage of reasoning in polyphonic audio, where multiple sound events co-occur and induce compositional structure. In this work, we introduce PolyBench, a benchmark designed to evaluate compositional reasoning in polyphonic audio. PolyBench comprises five evaluation subsets covering counting, classification, detection, concurrency, and duration estimation, requiring reasoning over multiple concurrent events and their relations. Evaluation of state-of-the-art LALMs reveals consistent performance degradation in polyphonic audio, indicating a fundamental bottleneck in current LALMs.

Problem

• compositional reasoning
• polyphonic audio
• audio language models
• benchmark
• concurrent sound events

Innovation

• compositional reasoning
• polyphonic audio
• audio language models
• benchmark
• concurrent sound events