Opening the Black Box: Preliminary Insights into Affective Modeling in Multimodal Foundation Models

2026-01-22

AI Summary

This study addresses the lack of interpretability in affective understanding and generation by multimodal foundation models. Through systematic investigation of their internal affective representation mechanisms, we uncover— for the first time—that affective adaptation is primarily localized in the gate projection (gate_proj) module of the feed-forward network, rather than in the attention mechanism. We rigorously validate the sufficiency, efficiency, and necessity of this module for affective modeling via module transfer, targeted fine-tuning, and destructive ablation across diverse architectures and tasks. Remarkably, fine-tuning only 24.5% of AffectGPT’s parameters achieves 96.6% of full-model performance on eight affective tasks, substantially improving parameter efficiency while maintaining high effectiveness.

Technology Category

• Cognitive Modeling & Cognitive Systems: Affective Computing
• Machine Learning: Multimodal Learning
• Natural Language Processing: Generation

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Understanding where and how emotions are represented in large-scale foundation models remains an open problem, particularly in multimodal affective settings. Despite the strong empirical performance of recent affective models, the internal architectural mechanisms that support affective understanding and generation are still poorly understood. In this work, we present a systematic mechanistic study of affective modeling in multimodal foundation models. Across multiple architectures, training strategies, and affective tasks, we analyze how emotion-oriented supervision reshapes internal model parameters. Our results consistently reveal a clear and robust pattern: affective adaptation does not primarily focus on the attention module, but instead localizes to the feed-forward gating projection (\texttt{gate\_proj}). Through controlled module transfer, targeted single-module adaptation, and destructive ablation, we further demonstrate that \texttt{gate\_proj} is sufficient, efficient, and necessary for affective understanding and generation. Notably, by tuning only approximately 24.5\% of the parameters tuned by AffectGPT, our approach achieves 96.6\% of its average performance across eight affective tasks, highlighting substantial parameter efficiency. Together, these findings provide empirical evidence that affective capabilities in foundation models are structurally mediated by feed-forward gating mechanisms and identify \texttt{gate\_proj} as a central architectural locus of affective modeling.

Problem

• affective modeling
• multimodal foundation models
• emotion representation
• model interpretability
• feed-forward gating

Innovation

• affective modeling
• multimodal foundation models
• gate_proj
• parameter efficiency
• mechanistic interpretability