Uncertainty Calibration of Multi-Label Bird Sound Classifiers

2025-11-11

AI Summary

This study systematically investigates uncertainty calibration in multi-label bird sound classifiers for bioacoustics, addressing real-world challenges including overlapping vocalizations, long-tailed species distributions, and train-deployment distribution shift. Leveraging the BirdSet benchmark, we conduct the first comprehensive calibration evaluation—global, dataset-level, and class-level—of four state-of-the-art models: Perch v2, ConvNeXt$_{BS}$, AudioProtoPNet, and BirdMAE. We employ a joint metric suite: Expected Calibration Error (ECE), Maximum Confidence Score (MCS, threshold-free), and class-mean Average Precision (cmAP, discriminative). Key findings reveal superior calibration for rare species; Perch v2 and ConvNeXt$_{BS}$ achieve best global calibration yet exhibit pervasive under-/over-confidence; and Platt scaling significantly improves calibration with minimal labeled data. Our work establishes a reproducible calibration evaluation framework and lightweight optimization pathway for trustworthy AI in bioacoustics.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Uncertainty Representations
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Security and Privacy: Large-scale security measurements

Abstract

Passive acoustic monitoring enables large-scale biodiversity assessment, but reliable classification of bioacoustic sounds requires not only high accuracy but also well-calibrated uncertainty estimates to ground decision-making. In bioacoustics, calibration is challenged by overlapping vocalisations, long-tailed species distributions, and distribution shifts between training and deployment data. The calibration of multi-label deep learning classifiers within the domain of bioacoustics has not yet been assessed. We systematically benchmark the calibration of four state-of-the-art multi-label bird sound classifiers on the BirdSet benchmark, evaluating both global, per-dataset and per-class calibration using threshold-free calibration metrics (ECE, MCS) alongside discrimination metrics (cmAP). Model calibration varies significantly across datasets and classes. While Perch v2 and ConvNeXt$_{BS}$ show better global calibration, results vary between datasets. Both models indicate consistent underconfidence, while AudioProtoPNet and BirdMAE are mostly overconfident. Surprisingly, calibration seems to be better for less frequent classes. Using simple post hoc calibration methods we demonstrate a straightforward way to improve calibration. A small labelled calibration set is sufficient to significantly improve calibration with Platt scaling, while global calibration parameters suffer from dataset variability. Our findings highlight the importance of evaluating and improving uncertainty calibration in bioacoustic classifiers.

Problem

• Calibrating uncertainty estimates for multi-label bird sound classifiers in bioacoustics
• Addressing calibration challenges from overlapping vocalizations and distribution shifts
• Improving reliability of biodiversity assessments through better uncertainty calibration

Innovation

• Multi-label bird sound classifiers calibration benchmark
• Post hoc Platt scaling improves uncertainty calibration
• Threshold-free metrics evaluate global and per-class calibration