Learning from Audio-Dependency Errors: Data Curation Strategies Based on Model Confusion Patterns in Audio Question Answering

2026-06-20

AI Summary

This work addresses the challenge in audio question answering where models often over-rely on textual priors or are misled by mismatched audio inputs. To mitigate this, the authors propose a diagnostic data curation strategy based on model confusion patterns. By probing model responses under normal, silent, and shuffled audio conditions, they identify and retain samples exhibiting strong audio dependence for fine-tuning. Using the Qwen3-Omni-30B-A3B-Instruct model, the approach integrates counterfactual audio testing, response normalization, and multi-model ensembling. Fine-tuning solely on the curated dataset yields a 67.27% accuracy on the official development set, significantly outperforming a local baseline of 65.90%, thereby demonstrating the method’s effectiveness in enhancing models’ reliance on genuine audio evidence.

Abstract

We frame the system as diagnostic data curation for a large audio-language model: before fine-tuning, we probe Qwen3-Omni-30B-A3B-Instruct under normal, empty-audio, and shuffled-audio conditions to identify how the model's answers change when audio evidence is removed or mismatched. These model confusion patterns are used to bucket training samples into text-prior, shuffle-leak, strong audio-dependent, and hard or misleading cases. Our strongest train-only system fine-tunes only on strong-audio items, where the normal audio-question pair is correct but both counterfactual variants fail, plus a small number of empty-audio negatives and a text-only response normalizer for parse-failed generations. On the official development set, the best train-only system reaches 67.27% accuracy after response normalization, compared with 65.90% for our local Qwen3-Omni baseline. Final submissions additionally include models trained using train+development splits and a three-model ensemble.

Problem

• audio question answering
• model confusion
• data curation
• audio dependency
• counterfactual evaluation

Innovation

• audio-language model
• diagnostic data curation
• model confusion patterns
• counterfactual probing
• audio dependency