Zero-Shot Respiratory Sound Classification through LLM-Augmented Audio-Text Alignment

2026-08-30

AI Summary

为解决自监督呼吸编码器缺乏临床语义基础的问题，提出一种框架通过医学大语言模型合成报告，并结合对比学习方法实现零样本分类。

Abstract

Self-supervised respiratory encoders lack semantic grounding in clinical domain needed for zero-shot inference, limiting their utility without task-specific labeled data. We propose a framework that aligns these encoders with medical terminology in a shared latent space turning them into a zero-shot-capable foundation model. To address paired data scarcity, we use a medical LLM to synthesize structured reports from metadata, creating dense semantic anchors for contrastive learning. Our training combines a sigmoid-based contrastive loss with encoder's native SSL objective and similarity-aware negative sampling to sharpen pathological boundaries. Across 9 tasks on 6 datasets, our method achieves a 61.3% mean zero-shot AUC, surpassing CLAP (51.4%) and Qwen2-Audio (54.9%) while reaching the highest linear probing AUC (71.6%) with only 43% of data used by full-scale baselines, showing that structured semantic alignment outperforms large-scale, general-purpose models in clinical diagnostics.

Problem

• self-supervised
• respiratory encoders
• semantic grounding
• zero-shot inference
• clinical domain

Innovation

• Zero-Shot Classification
• LLM-Augmented Audio-Text Alignment
• Contrastive Learning
• Semantic Anchors
• Similarity-Aware Negative Sampling