🤖 AI Summary
Small-scale and low-quality breath sound datasets hinder classification performance, while ensemble models—though effective—impose substantial inference overhead. To address this, we propose an architecture-agnostic soft-label knowledge distillation framework that efficiently transfers knowledge from multiple teachers to a lightweight student model. Our key contributions are threefold: (1) We empirically demonstrate for the first time that a single teacher—architecturally identical to the student—suffices to yield significant performance gains; (2) we show that only a few teachers are needed to achieve near-optimal improvement, thereby relaxing the conventional requirement of heterogeneous, high-capacity teachers; and (3) we integrate soft-label distillation, ensemble learning, and time-frequency feature modeling tailored to respiratory acoustics. Evaluated on the ICHBI dataset, our method achieves a new state-of-the-art score of 64.39 (+0.85), with an average architecture-wise performance gain exceeding 1.16.
📝 Abstract
Respiratory sound datasets are limited in size and quality, making high performance difficult to achieve. Ensemble models help but inevitably increase compute cost at inference time. Soft label training distills knowledge efficiently with extra cost only at training. In this study, we explore soft labels for respiratory sound classification as an architecture-agnostic approach to distill an ensemble of teacher models into a student model. We examine different variations of our approach and find that even a single teacher, identical to the student, considerably improves performance beyond its own capability, with optimal gains achieved using only a few teachers. We achieve the new state-of-the-art Score of 64.39 on ICHBI, surpassing the previous best by 0.85 and improving average Scores across architectures by more than 1.16. Our results highlight the effectiveness of knowledge distillation with soft labels for respiratory sound classification, regardless of size or architecture.