Do We Really Need Multimodal Emotion Language Models Larger Than 1B Parameters?

2026-07-14

AI Summary

This work addresses the challenge of deploying large-scale multimodal emotion recognition (MER) models on resource-constrained devices due to their high parameter counts and computational costs. To this end, the authors propose Light-MER, a lightweight framework that leverages knowledge distillation to transfer capabilities from large teacher models to student models with fewer than one billion parameters. The approach innovatively integrates sliced Wasserstein distance with optimal transport–based loss for aligning hidden states and introduces a GRPO-based multi-reward optimization strategy to enhance knowledge transfer efficacy. Evaluated across nine benchmark datasets, Light-MER achieves state-of-the-art performance while substantially reducing inference costs, demonstrating the feasibility and practical value of efficient multimodal emotion recognition.

Abstract

Recent advances in multimodal large language models (MLLMs) have significantly improved the performance of multimodal emotion recognition (MER) and enabled interpretable description generation by jointly modeling video, audio, and language, etc. However, these performance improvements are often accompanied by an increase in model parameter size (e.g, at least 7B), which simultaneously incurs high computational costs and reduces inference efficiency, thereby hindering real-time deployment on resource-constrained platforms such as robots and mobile devices. This raises a fundamental question: do we really need the multimodal MER model larger than 1B parameters for high-quality MER? In this paper, we challenge the assumption that larger models are inherently necessary and proposes a lightweight MER framework (called Light-MER), which achieves better and faster multimodal sentiment understanding and recognition through knowledge distillation. It can transfer knowledge from a strong, large-scale teacher model to a lightweight sub-billion-parameter student model, aiming to preserve rich multimodal emotion reasoning and recognition while substantially improving deployment efficiency. Specifically, we introduce two new optimization strategies to enhance knowledge transfer: (1) a new optimal transport loss that combines Sliced Wasserstein Distance with hidden-state alignment, and (2) a new multi-reward optimization strategy based on GRPO that balances MER performance and efficiency, aimed at further enhancing the learning capabilities of student models. Extensive experiments on nine benchmark datasets demonstrate that Light-MER achieves state-of-the-art performance while significantly improving inference efficiency. This highlights the strong potential of small multimodal emotion language models for future research. Code is available at https://github.com/GAIR-Lab/Light-MER.

Problem

• multimodal emotion recognition
• large language models
• model efficiency
• parameter size
• real-time deployment

Innovation

• knowledge distillation
• lightweight multimodal model
• optimal transport
• multi-reward optimization
• multimodal emotion recognition