RA-QA: A Benchmarking System for Respiratory Audio Question Answering Under Real-World Heterogeneity

2026-02-04

AI Summary

Existing research on respiratory sound question answering lacks systematic evaluation of the heterogeneity inherent in real-world settings—spanning multiple modalities, recording devices, and question types—making it difficult to assess model robustness in clinical scenarios. To address this gap, this work introduces RA-QA, the first benchmark specifically designed for respiratory audio question answering under realistic heterogeneous conditions. RA-QA integrates multiple public datasets and employs a standardized pipeline to generate 9 million diverse question-answer pairs encompassing diagnostic and contextual attributes. The benchmark establishes a unified evaluation protocol that enables comprehensive assessment across modalities, devices, and question types, providing reproducible baselines for both general-purpose audio-language models and domain-specific architectures, thereby revealing critical performance limitations of current approaches under real-world heterogeneity.

Abstract

As conversational multimodal AI tools are increasingly adopted to process patient data for health assessment, robust benchmarks are needed to measure progress and expose failure modes under realistic conditions. Despite the importance of respiratory audio for mobile health screening, respiratory audio question answering remains underexplored, with existing studies evaluated narrowly and lacking real-world heterogeneity across modalities, devices, and question types. We hence introduce the Respiratory-Audio Question-Answering (RA-QA) benchmark, including a standardized data generation pipeline, a comprehensive multimodal QA collection, and a unified evaluation protocol. RA-QA harmonizes public RA datasets into a collection of 9 million format-diverse QA pairs covering diagnostic and contextual attributes. We benchmark classical ML baselines alongside multimodal audio-language models, establishing reproducible reference points and showing how current approaches fail under heterogeneity.

Problem

• respiratory audio
• question answering
• real-world heterogeneity
• multimodal AI
• benchmarking

Innovation

• Respiratory Audio
• Question Answering
• Multimodal Benchmark
• Real-World Heterogeneity
• Audio-Language Models