Acoustic Prompting via Stage-wise Modulation for Few-Shot Learning in Audio Language Models

2026-06-14

AI Summary

This work addresses the limitation of existing audio language models in few-shot learning, which typically optimize only textual prompts while overlooking the potential of learnable prompts within the audio encoder. The study introduces trainable prompts directly into the audio encoder for the first time, employing a staged modulation mechanism to explicitly guide the learning of task-relevant acoustic features. These audio-side prompts are jointly optimized with textual prompts, forming a complementary bilateral prompting architecture. Implemented in a plug-and-play manner, the proposed method is seamlessly integrated into existing models and achieves substantial improvements in few-shot performance across eleven audio classification benchmarks, demonstrating the effectiveness and generalizability of audio-side prompting in modulating the representation space.

Abstract

Audio-Language Models (ALMs) have shown remarkable success in zero-shot audio classification by aligning audio waveforms with text. Recent efforts to improve downstream performance focus on learning optimal text prompts. However, previous approaches focus on the text encoder, leaving the potential of learnable prompts within the audio encoder unexplored. In this paper, we propose a novel framework that introduces trainable prompts into the audio encoder to capture task-specific acoustic features. We demonstrate that integrating audio-side prompt learning with existing text-side approaches enhances few-shot adaptation. Through extensive experiments across 11 datasets show that integrating our method as a plug-and-play module alongside existing text prompt tuning generally leads to performance improvements. These findings suggest that explicitly modulating the audio representation space effectively complements text-only prompting approaches. The code is available at https://github.com/hyebin-c/aspl.

Problem

• Audio-Language Models
• Few-Shot Learning
• Prompt Tuning
• Acoustic Features
• Audio Encoder

Innovation

• Acoustic Prompting
• Stage-wise Modulation
• Few-Shot Learning
• Audio-Language Models
• Prompt Tuning