ParaPairAudioBench: Paralinguistic Pairwise Audio Benchmark for LALM-as-a-Judge

2026-06-23

AI Summary

This work addresses the limitations of current large audio language models (LALMs) as evaluators of speech, which predominantly focus on overall naturalness and struggle to discern fine-grained paralinguistic attributes. To this end, we introduce the first paired audio evaluation benchmark comprising 5,175 sample pairs annotated across five dimensions—style, speaking rate, emphasis, age, and gender—enabling multidimensional assessment under both same-text and cross-text conditions. We further propose a calibration-aware mechanism to enhance model reliability in scenarios where abstention is warranted. Experimental results reveal that existing LALMs exhibit an average accuracy 32 percentage points lower than human evaluators in paralinguistic judgment and suffer from significant miscalibration, particularly in tie cases.

Abstract

Large Audio-Language Models (LALMs) have been widely used as judge models for the automatic evaluation of generated speech. However, prior approaches predominantly focus on holistic naturalness, leaving fine-grained paralinguistic distinctions underexplored. We introduce ParaPairAudioBench, a pairwise benchmark of 5,175 audio pairs across five paralinguistic dimensions: Style, Rate, Emphasis, Age, and Gender. Our experiments show that current LALM judges still lag behind human judgments by 32%p on average and exhibit severe calibration failures, particularly in Tie cases where the correct decision is to abstain. To further analyze lexical versus acoustic reliance, the benchmark includes both same-transcript and cross-transcript conditions. ParaPairAudioBench enables multi-dimensional, calibration-aware assessment of the reliability of LALM-as-a-Judge for paralinguistic speech evaluation.

Problem

• paralinguistic
• audio-language models
• speech evaluation
• pairwise benchmark
• calibration

Innovation

• Paralinguistic
• Pairwise Benchmark
• LALM-as-a-Judge
• Calibration Failure
• Audio-Language Model