Auditing Protocol-Level Shortcuts in Large Audio Language Model Judges for Speech Evaluation

2026-07-15

AI Summary

This study addresses a critical yet overlooked issue in evaluating large audio language models: their potential reliance on cues from evaluation protocols—such as provided labels or reference information—rather than genuine comprehension of audio content, which can artificially inflate human-model agreement. To probe such protocol-level shortcuts, the authors introduce a novel auditing framework employing three adversarial manipulations: feature blueprint substitution, reference information perturbation, and option order swapping. Applying this approach across six prominent models and four speech attributes, the experiments reveal substantial shortcut dependence; for instance, emotion recognition accuracy drops below 0.10 for several models, and Qwen3-Omni-Thinking consistently selects the same position in A/B tests regardless of content. These findings underscore the necessity of jointly assessing both model capabilities and the validity of evaluation protocols themselves.

Abstract

Large audio-language models (LALMs) are increasingly used as automatic judges for speech evaluation. However, high agreement with human ratings does not guarantee that their verdicts are grounded in the audio. A judge may instead rely on specialist labels or reference data supplied by the evaluation protocol itself, taking a shortcut in place of listening to the audio. In this paper, we audit such protocol-level ``shortcuts'' in LALM judges across three common deployment protocols: feature-blueprint judging, where the audio is replaced by a structured text description of acoustic features, reference-conditioned judging, and pairwise A/B comparison. Across six judges and four attributes, we find that several LALMs rely on protocol-level shortcuts. For example, in feature-blueprint judging, incorrect specialist labels reduce five judges' emotion accuracy to 0.10 or below, and in concatenated A/B comparisons, Qwen3-Omni-Thinking often picks the same slot regardless of order swaps. These results indicate that aggregate agreement can overstate the validity of LALM judges unless the model and the evaluation protocol are assessed jointly, and that each model-protocol pair should be evaluated with a matched shortcut probe.

Problem

• audio-language models
• speech evaluation
• protocol-level shortcuts
• automatic judging
• evaluation validity

Innovation

• protocol-level shortcuts
• large audio-language models
• speech evaluation
• shortcut auditing
• evaluation protocol