ZEBRA: Zero-Shot Entropy-Regularized Prompt Learning for Base-to-Novel Generalization in Audio-Language Models

2026-06-30

AI Summary

This work addresses the generalization gap in audio-language models during prompt learning, where improved performance on base classes often comes at the expense of degraded zero-shot recognition accuracy on novel classes. To mitigate this trade-off, the paper proposes ZEBRA, a novel framework that introduces self-entropy regularization into prompt learning for the first time, effectively suppressing overfitting to base classes. Additionally, ZEBRA enhances audio-language alignment by fusing logits from both zero-shot inference and prompt-based learning. Evaluated across multiple audio classification benchmarks, the method significantly improves zero-shot recognition accuracy on novel classes while maintaining strong performance on base classes, thereby substantially narrowing the generalization gap between them.

Abstract

Audio-Language Models (ALMs) achieve strong zero-shot performance by aligning audio with textual class descriptions. Although prompt learning improves accuracy on base classes through few-shot supervised adaptation, we observe a critical trade-off: it often degrades performance on novel classes, sometimes falling below zero-shot accuracy. This exposes a base-to-novel generalization gap in prompt learning for ALMs. To address this issue, we propose \textbf{ZEBRA} (Zero-shot Entropy-Regularized Prompt Learning for Base-to-Novel Generalization), a plug-and-play framework that fuses zero-shot logits with prompt-learning logits, and employs self-entropy regularization to reduce overfitting to base classes. Experiments across multiple audio classification datasets show that ZEBRA consistently improves novel-class performance while maintaining strong base accuracy, significantly reducing the base-to-novel gap compared to standard prompt learning. The code is available at: https://github.com/asif-hanif/zebra.

Problem

• Audio-Language Models
• prompt learning
• zero-shot
• base-to-novel generalization
• generalization gap

Innovation

• zero-shot learning
• prompt learning
• entropy regularization
• audio-language models
• base-to-novel generalization