Unlocking In-Context Learning in Audio-Language Models from Decentralized Medical Audio

2026-06-22

AI Summary

This work addresses the dual challenges of scarce labeled data and privacy constraints in resource-limited clinical settings for medical audio diagnosis. It proposes a Federated Self-Contextualization (FSC) framework that, for the first time, integrates in-context learning into decentralized medical audio analysis. By leveraging unsupervised clustering to generate pseudo-labeled segments and combining audio-language multimodal modeling, caption-guided pretraining, and a support-query inference mechanism within a federated learning paradigm, the method enables few-shot contextual diagnosis without requiring ground-truth annotations. Evaluated on 2-way 2-shot tasks for respiratory and cardiac disease classification, the approach achieves 71.6% accuracy—surpassing existing audio-language baselines by over 9%—thereby substantially advancing the feasibility and performance of low-resource medical audio analysis.

Abstract

Clinical audio diagnosis in low-resource settings requires models that identify conditions from minimal examples without large annotated corpora. We propose Federated Self-Contextualization (FSC), a multimodal language model framework for in-context clinical audio diagnosis across federated hospital clients. FSC constructs pseudo-label episodes via unsupervised clustering of audio representations, bypassing scarce real diagnostic labels, and enables contextual reasoning from support-query pairs. Our progressive three-stage pipeline first aligns audio embeddings with the language model via caption-based pretraining, then adapts it for episodic in-context inference through federated optimization. At test time, given a small labeled support set, the model diagnoses an unseen query through multimodal reasoning. On held-out respiratory and cardiac conditions, FSC achieves 71.6% accuracy in 2-way 2-shot evaluation, outperforming audio-language baselines by over 9%.

Problem

• in-context learning
• audio-language models
• federated learning
• low-resource settings
• clinical audio diagnosis

Innovation

• Federated Self-Contextualization
• in-context learning
• audio-language models
• unsupervised pseudo-labeling
• federated multimodal learning