A Multimodal-Multitask Framework with Cross-modal Relation and Hierarchical Interactive Attention for Semantic Comprehension

2025-08-22

AI Summary

Multimodal learning faces two key challenges: intra-modal noise corrupting joint representations and loss of modality-specific discriminative information during fusion. To address these, we propose a multi-task multimodal semantic understanding framework. Our method employs (1) a cross-modal relational graph that implicitly models inter-modal dependencies—thereby avoiding explicit interaction and mitigating noise propagation—and (2) a Hierarchical Interactive Modality-wise Attention (HIMA) mechanism that enhances intra-modal feature extraction and discriminative modeling prior to late fusion. By integrating graph neural networks with attention mechanisms, the framework enables neighborhood-driven feature reconstruction and fine-grained semantic modeling. Extensive experiments on three public benchmarks demonstrate substantial improvements in both accuracy and robustness over state-of-the-art methods, validating the effectiveness of our approach in preserving modality-specific semantics while enabling robust cross-modal integration.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

A major challenge in multimodal learning is the presence of noise within individual modalities. This noise inherently affects the resulting multimodal representations, especially when these representations are obtained through explicit interactions between different modalities. Moreover, the multimodal fusion techniques while aiming to achieve a strong joint representation, can neglect valuable discriminative information within the individual modalities. To this end, we propose a Multimodal-Multitask framework with crOss-modal Relation and hIErarchical iNteractive aTtention (MM-ORIENT) that is effective for multiple tasks. The proposed approach acquires multimodal representations cross-modally without explicit interaction between different modalities, reducing the noise effect at the latent stage. To achieve this, we propose cross-modal relation graphs that reconstruct monomodal features to acquire multimodal representations. The features are reconstructed based on the node neighborhood, where the neighborhood is decided by the features of a different modality. We also propose Hierarchical Interactive Monomadal Attention (HIMA) to focus on pertinent information within a modality. While cross-modal relation graphs help comprehend high-order relationships between two modalities, HIMA helps in multitasking by learning discriminative features of individual modalities before late-fusing them. Finally, extensive experimental evaluation on three datasets demonstrates that the proposed approach effectively comprehends multimodal content for multiple tasks.

Problem

• Reducing noise in multimodal representations
• Preserving discriminative information within individual modalities
• Enhancing semantic comprehension across multiple tasks

Innovation

• Cross-modal relation graphs reconstruct monomodal features
• Hierarchical Interactive Attention focuses on pertinent information
• Multimodal-Multitask framework without explicit modality interaction