A Toolkit for Detecting Spurious Correlations in Speech Datasets

2026-04-29

AI Summary

This work addresses the risk of spurious correlations between recording artifacts and target labels in speech datasets, which arise from heterogeneous recording conditions and can lead to overestimated model performance—posing significant safety concerns in high-stakes domains such as healthcare. To detect such confounding factors, the study introduces a novel diagnostic method that leverages non-speech segments within audio recordings to predict target labels, exploiting the metadata implicitly embedded in these silent or non-linguistic regions. By analyzing these non-speech intervals, the approach effectively identifies spurious associations present in both training and test data. The authors further release an open-source toolkit implementing this technique, substantially enhancing the reliability of speech model evaluation and the trustworthiness of real-world deployment.

Abstract

We introduce a toolkit for uncovering spurious correlations between recording characteristics and target class in speech datasets. Spurious correlations may arise due to heterogeneous recording conditions, a common scenario for health-related datasets. When present both in the training and test data, these correlations result in an overestimation of the system performance -- a dangerous situation, specially in high-stakes application where systems are required to satisfy minimum performance requirements. Our toolkit implements a diagnostic method based on the detection of the target class using only the non-speech regions in the audio. Better than chance performance at this task indicates that information about the target class can be extracted from the non-speech regions, flagging the presence of spurious correlations. The toolkit is publicly available for research use.

Problem

• spurious correlations
• speech datasets
• recording conditions
• performance overestimation
• non-speech regions

Innovation

• spurious correlations
• speech datasets
• non-speech regions
• diagnostic method
• recording conditions