AudioCapBench: Quick Evaluation on Audio Captioning across Sound, Music, and Speech

2026-02-27

AI Summary

This study addresses the lack of a unified and efficient benchmark for evaluating large models’ ability to generate descriptions across the three major audio domains: sound, music, and speech. The authors present the first comprehensive evaluation benchmark encompassing all three categories, comprising 1,000 carefully curated samples, and introduce a fine-grained assessment framework along three orthogonal dimensions: accuracy, completeness, and hallucination. Combining traditional metrics (METEOR, BLEU, ROUGE-L) with an LLM-as-Judge approach based on large language models, they conduct an automated yet human-aligned evaluation of 13 prominent multimodal models, including OpenAI’s and Gemini series. Results indicate that the Gemini series achieves overall superior performance (Gemini 3 Pro scores 6.00/10), while OpenAI models exhibit fewer hallucinations; all models perform best on speech-related tasks and struggle most with music.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

We introduce AudioCapBench, a benchmark for evaluating audio captioning capabilities of large multimodal models. \method covers three distinct audio domains, including environmental sound, music, and speech, with 1,000 curated evaluation samples drawn from established datasets. We evaluate 13 models across two providers (OpenAI, Google Gemini) using both reference-based metrics (METEOR, BLEU, ROUGE-L) and an LLM-as-Judge framework that scores predictions on three orthogonal dimensions: \textit{accuracy} (semantic correctness), \textit{completeness} (coverage of reference content), and \textit{hallucination} (absence of fabricated content). Our results reveal that Gemini models generally outperform OpenAI models on overall captioning quality, with Gemini~3~Pro achieving the highest overall score (6.00/10), while OpenAI models exhibit lower hallucination rates. All models perform best on speech captioning and worst on music captioning. We release the benchmark as well as evaluation code to facilitate reproducible audio understanding research.

Problem

• audio captioning
• multimodal models
• benchmark evaluation
• music
• speech

Innovation

• Audio Captioning
• Multimodal Benchmark
• LLM-as-Judge
• Cross-domain Evaluation
• Hallucination Assessment