ORCA: Open-ended Response Correctness Assessment for Audio Question Answering

2025-11-28

AI Summary

Evaluating open-ended audio question answering (QA) for large audio-language models (LALMs) faces challenges including high inter-annotator disagreement, ambiguity in partial correctness, and the inadequacy of scalar scores to capture uncertainty. Method: We propose the first automated evaluation framework that explicitly models judgment uncertainty. Specifically, (1) we introduce Beta-distributed modeling of answer correctness—jointly estimating expected correctness and its uncertainty—and (2) design a three-stage human-in-the-loop annotation paradigm integrating structured human feedback with iterative refinement. Results: Evaluated on 3,580 audio QA pairs, our framework achieves Krippendorff’s alpha = 0.82 and Spearman correlation = 0.91—significantly outperforming LLM-based judges—while reducing computational overhead substantially. This work establishes a new, interpretable, robust, and low-resource paradigm for evaluating open-generative audio understanding.

Technology Category

• Natural Language Processing: Question Answering
• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Uncertainty Representations

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Evaluating open-ended responses from large audio language models (LALMs) is challenging because human annotators often genuinely disagree on answer correctness due to multiple valid interpretations, partial correctness, and subjective judgment. Traditional metrics reporting only mean scores fail to capture this uncertainty. We present ORCA (Open-ended Response Correctness Assessment), a framework that models the variability in human judgments using Beta distributions to predict both expected correctness and uncertainty. Our three-stage annotation framework combines human judgment with structured feedback and iterative refinement to simultaneously curate training data and improve benchmark quality. We collected 11,721 annotations across 3,580 question-answer pairs from 15 LALMs on two audio QA benchmarks, achieving inter-annotator agreement of 0.82 (Krippendorff's alpha). ORCA achieves 0.91 Spearman correlation with mean human judgments, matching or outperforming LLM-judge baselines while providing uncertainty estimates and requiring significantly less compute. We release our models, code, and curated dataset.

Problem

• Assessing correctness of open-ended audio QA responses with human disagreement.
• Modeling judgment variability and uncertainty using Beta distributions.
• Creating a framework to improve benchmark quality and training data.

Innovation

• Beta distributions model human judgment variability
• Three-stage annotation combines human and structured feedback
• Achieves high correlation with human judgments efficiently