ABHINAYA -- A System for Speech Emotion Recognition In Naturalistic Conditions Challenge

2025-05-23

AI Summary

Natural speech emotion recognition (SER) faces fundamental challenges in real-world settings, including inherent speaker and contextual variability, diverse recording conditions, and severe class imbalance. To address these issues, this paper proposes a speech-text multimodal joint modeling framework. Our approach introduces a novel speech-large language model (SLLM)-driven cross-modal alignment mechanism that synergistically integrates self-supervised speech representations with large language model (LLM)-based textual understanding for fine-grained emotional cue extraction. Additionally, we design a class-weighted loss function and an ensemble-based majority voting decision strategy to mitigate class imbalance. Evaluated on the Interspeech Naturalistic SER Challenge, our method ranks 4th among 166 participating teams and achieves state-of-the-art (SOTA) performance after full training. The framework demonstrates significantly improved robustness and generalization capability in realistic, unconstrained environments.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Speech emotion recognition (SER) in naturalistic settings remains a challenge due to the intrinsic variability, diverse recording conditions, and class imbalance. As participants in the Interspeech Naturalistic SER Challenge which focused on these complexities, we present Abhinaya, a system integrating speech-based, text-based, and speech-text models. Our approach fine-tunes self-supervised and speech large language models (SLLM) for speech representations, leverages large language models (LLM) for textual context, and employs speech-text modeling with an SLLM to capture nuanced emotional cues. To combat class imbalance, we apply tailored loss functions and generate categorical decisions through majority voting. Despite one model not being fully trained, the Abhinaya system ranked 4th among 166 submissions. Upon completion of training, it achieved state-of-the-art performance among published results, demonstrating the effectiveness of our approach for SER in real-world conditions.

Problem

• Speech emotion recognition in naturalistic settings is challenging due to variability and class imbalance.
• The paper presents Abhinaya, integrating speech, text, and speech-text models for SER.
• The system addresses class imbalance with tailored loss functions and majority voting.

Innovation

• Fine-tunes self-supervised and speech LLMs
• Leverages LLMs for textual context
• Employs speech-text modeling with SLLM