CD-MED: Cross-Domain Multimodal Emotion Descriptor for Visual Comparison of Digital Objects

2026-07-14

AI Summary

This work addresses the limitation of existing emotion recognition models, which are typically confined to a single modality and thus unable to support cross-domain affective comparison across heterogeneous digital objects such as films, music, images, and text. To overcome this challenge, the authors propose CD-MED, a novel framework that establishes a unified cross-domain multimodal emotional representation. The method employs modality-specific models to extract affective features, which are then fused and mapped into a two-dimensional valence–arousal affective space. This representation is further enhanced through multidimensional visual encoding—incorporating position, color, size, and shape—to enable consistent and interpretable descriptions of emotions across media types. The resulting framework facilitates visual comparison, retrieval, and recommendation of cross-media emotional content, significantly improving both consistency and interpretability in multimodal affective expression.

Abstract

Digital objects express emotions through different modalities. For example, a movie may include visual scenes, audio, dialogue, and facial expressions, while a song may contain melody, rhythm, lyrics, and vocal tone. Because existing emotion recognition models are usually modality-specific, it is difficult to compare such objects directly. This paper proposes CD-MED, a Cross-Domain Multimodal Emotion Descriptor for representing heterogeneous digital objects in a common emotional space. Each modality can be processed by its own emotion recognition model, and the resulting emotional outputs are transformed into a shared descriptor. The descriptor preserves information from individual modalities while also allowing an integrated emotional profile of the object. For interpretation, CD-MED is visualized in the valence-arousal space: position represents affective coordinates, color denotes emotion category, size indicates intensity, and shape shows the modality. This unified representation enables emotion-based comparison, retrieval, recommendation, and visualization across different domains such as movies, songs, images, and books.

Problem

• cross-domain
• multimodal emotion
• emotion comparison
• heterogeneous digital objects
• emotion recognition

Innovation

• Cross-Domain
• Multimodal Emotion Representation
• Emotion Descriptor
• Valence-Arousal Space
• Digital Object Comparison