Benchmarking and Confidence Evaluation of LALMs For Temporal Reasoning

2025-05-19

AI Summary

Large Audio-Language Models (LALMs) lack standardized evaluation for temporal reasoning tasks, hindering assessment of their reliability and trustworthiness. Method: We introduce TREA, the first benchmark dedicated to temporal reasoning for LALMs, featuring semantically invariant input perturbations and a unified uncertainty quantification framework to systematically evaluate open-source LALMs across multi-granularity temporal understanding tasks. Results: Current LALMs substantially underperform human-level temporal reasoning; their prediction accuracy exhibits negligible correlation with confidence scores—indicating pervasive “overconfidence”; and our proposed uncertainty metrics effectively identify high-risk mispredictions. This work shifts LALM evaluation from accuracy-centric paradigms toward a new standard emphasizing both reliability and robustness, providing interpretable, calibratable assessment foundations for safety-critical applications such as healthcare and security monitoring.

Technology Category

• Reasoning under Uncertainty: Uncertainty Representations
• Machine Learning: Calibration & Uncertainty Quantification
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

The popular success of text-based large language models (LLM) has streamlined the attention of the multimodal community to combine other modalities like vision and audio along with text to achieve similar multimodal capabilities. In this quest, large audio language models (LALMs) have to be evaluated on reasoning related tasks which are different from traditional classification or generation tasks. Towards this goal, we propose a novel dataset called temporal reasoning evaluation of audio (TREA). We benchmark open-source LALMs and observe that they are consistently behind human capabilities on the tasks in the TREA dataset. While evaluating LALMs, we also propose an uncertainty metric, which computes the invariance of the model to semantically identical perturbations of the input. Our analysis shows that the accuracy and uncertainty metrics are not necessarily correlated and thus, points to a need for wholesome evaluation of LALMs for high-stakes applications.

Problem

• Evaluating LALMs' temporal reasoning vs human performance
• Assessing LALMs' uncertainty via input perturbation invariance
• Identifying accuracy-uncertainty gap for high-stakes LALM evaluation

Innovation

• Proposed TREA dataset for temporal reasoning evaluation
• Benchmarked LALMs against human performance metrics
• Introduced uncertainty metric for model invariance assessment