FinAudio: A Benchmark for Audio Large Language Models in Financial Applications

2025-03-26

AI Summary

Existing AudioLLMs lack systematic evaluation benchmarks tailored to financial scenarios (e.g., earnings calls, CEO speeches). This paper introduces FinAudio—the first dedicated benchmark for financial audio understanding—comprising three tasks: short- and long-audio ASR, and long-audio summarization. Its contributions are threefold: (1) it formally defines domain-specific audio understanding tasks for finance; (2) it constructs the first high-quality financial audio dataset, featuring multi-stage cleaning, domain-aligned transcription, expert-validated annotations, and original abstractive summaries; and (3) it establishes standardized annotation guidelines and an open-source evaluation framework. Zero-shot and fine-tuned evaluations across seven state-of-the-art AudioLLMs reveal substantial performance degradation on long-audio ASR and financial summarization—ROUGE-L drops by 18.3% on average—exposing critical limitations in professional speech comprehension. All data, annotation protocols, and code are publicly released.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Summarization
• Cognitive Modeling & Cognitive Systems: Affective Computing

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Audio Large Language Models (AudioLLMs) have received widespread attention and have significantly improved performance on audio tasks such as conversation, audio understanding, and automatic speech recognition (ASR). Despite these advancements, there is an absence of a benchmark for assessing AudioLLMs in financial scenarios, where audio data, such as earnings conference calls and CEO speeches, are crucial resources for financial analysis and investment decisions. In this paper, we introduce extsc{FinAudio}, the first benchmark designed to evaluate the capacity of AudioLLMs in the financial domain. We first define three tasks based on the unique characteristics of the financial domain: 1) ASR for short financial audio, 2) ASR for long financial audio, and 3) summarization of long financial audio. Then, we curate two short and two long audio datasets, respectively, and develop a novel dataset for financial audio summarization, comprising the extsc{FinAudio} benchmark. Then, we evaluate seven prevalent AudioLLMs on extsc{FinAudio}. Our evaluation reveals the limitations of existing AudioLLMs in the financial domain and offers insights for improving AudioLLMs. All datasets and codes will be released.

Problem

• Lack of financial benchmark for AudioLLMs evaluation
• Need for ASR in short and long financial audio
• Requirement for summarization of financial audio data

Innovation

• First benchmark for financial AudioLLMs evaluation
• Defines three financial-specific audio tasks
• Introduces novel financial audio summarization dataset