Robult: Leveraging Redundancy and Modality Specific Features for Robust Multimodal Learning

2025-09-03

AI Summary

To address the degraded robustness in multimodal learning caused by modality missing and scarce labeled data, this paper proposes a lightweight, scalable semi-supervised framework. Methodologically, it innovatively combines soft positive–unlabeled contrastive loss with latent-space reconstruction loss to achieve task-oriented cross-modal feature alignment while preserving modality-specific representations. Information-theoretic constraints and a modular architecture further enhance generalizability and enable seamless integration with mainstream models. Extensive experiments on multiple benchmark datasets demonstrate that the method significantly outperforms existing approaches under random modality missing and low-label-rate settings. It achieves high accuracy, strong robustness to modality corruption, and low computational overhead—making it suitable for real-world deployment.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Addressing missing modalities and limited labeled data is crucial for advancing robust multimodal learning. We propose Robult, a scalable framework designed to mitigate these challenges by preserving modality-specific information and leveraging redundancy through a novel information-theoretic approach. Robult optimizes two core objectives: (1) a soft Positive-Unlabeled (PU) contrastive loss that maximizes task-relevant feature alignment while effectively utilizing limited labeled data in semi-supervised settings, and (2) a latent reconstruction loss that ensures unique modality-specific information is retained. These strategies, embedded within a modular design, enhance performance across various downstream tasks and ensure resilience to incomplete modalities during inference. Experimental results across diverse datasets validate that Robult achieves superior performance over existing approaches in both semi-supervised learning and missing modality contexts. Furthermore, its lightweight design promotes scalability and seamless integration with existing architectures, making it suitable for real-world multimodal applications.

Problem

• Addressing missing modalities in multimodal learning
• Leveraging limited labeled data for robustness
• Preserving modality-specific features using information theory

Innovation

• Soft PU contrastive loss for semi-supervised alignment
• Latent reconstruction loss preserves modality-specific information
• Modular lightweight design ensures scalability and integration