Prompt Amplification and Zero-Shot Late Fusion in Audio-Language Models for Speech Emotion Recognition

2026-03-24

AI Summary

This work addresses the insufficient synergy between general-purpose audio language models and domain-specific architectures in zero-shot speech emotion recognition. To this end, we propose ZS-Fuse, a novel approach that, for the first time, integrates emotion predictions from a dual-encoder audio language model with outputs from a specialized foundation model through a late-fusion strategy under a zero-shot setting. Our method innovatively incorporates prompt ensembling and prompt amplification techniques to enhance model robustness and unlock stronger zero-shot capabilities. Extensive experiments demonstrate that ZS-Fuse consistently outperforms current state-of-the-art methods—including WavLM-Large—across three standard speech emotion recognition benchmarks, thereby validating its effectiveness and generalization performance.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Audio-Language Models (ALMs) are making strides in understanding speech and non-speech audio. However, domain-specialist Foundation Models (FMs) remain the best for closed-ended speech processing tasks such as Speech Emotion Recognition (SER). Using ALMs for Zero-shot SER is a popular choice, but their potential to work with specialists to achieve state-of-the-art (SOTA) performance remains unexplored. We propose ZS-Fuse, a late-fusion method that combines zero-shot emotion estimates from a dual-encoder ALM with specialist FMs. To handle ambiguity in emotions and sensitivity to prompt choice, 1) we use a simple prompt ensemble and 2) suggest a novel technique called prompt amplification, which repeats audio and text queries to discover stronger zero-shot capabilities. We demonstrate the efficacy of our technique by evaluating ZS-Fuse with three dual-encoder ALMs and two FMs, and report improvements over SOTA baselines, such as WavLM-Large, on three speech emotion recognition datasets.

Problem

• Speech Emotion Recognition
• Audio-Language Models
• Zero-Shot Learning
• Prompt Sensitivity
• Emotion Ambiguity

Innovation

• Prompt Amplification
• Zero-Shot Late Fusion
• Audio-Language Models
• Speech Emotion Recognition
• Foundation Models