FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion

2024-02-05

Neural Information Processing Systems

AI Summary

This paper addresses three pervasive challenges in multimodal time-series data: modality missingness, irregular sampling, and scarcity of labeled examples. To this end, we propose a scalable Mixture-of-Experts (MoE) architecture. Our method integrates a Transformer-based backbone with modality-agnostic feature alignment and a missingness-aware routing strategy to enable robust cross-modal fusion. Crucially, we introduce the first dynamic sparse gating mechanism explicitly designed for incomplete and asynchronous multimodal inputs—guaranteed to converge theoretically and supporting arbitrary subsets of available modalities. Experimental results across multiple real-world time-series forecasting benchmarks demonstrate that our approach significantly outperforms existing state-of-the-art methods. Notably, it maintains strong generalization and stability even under extreme conditions: modality missing rates exceeding 60% and highly irregular sampling patterns.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs

Abstract

As machine learning models in critical fields increasingly grapple with multimodal data, they face the dual challenges of handling a wide array of modalities, often incomplete due to missing elements, and the temporal irregularity and sparsity of collected samples. Successfully leveraging this complex data, while overcoming the scarcity of high-quality training samples, is key to improving these models' predictive performance. We introduce ``FuseMoE'', a mixture-of-experts framework incorporated with an innovative gating function. Designed to integrate a diverse number of modalities, FuseMoE is effective in managing scenarios with missing modalities and irregularly sampled data trajectories. Theoretically, our unique gating function contributes to enhanced convergence rates, leading to better performance in multiple downstream tasks. The practical utility of FuseMoE in the real world is validated by a diverse set of challenging prediction tasks.

Problem

• Handling incomplete multimodal data with missing elements
• Managing temporal irregularity and sparsity in samples
• Overcoming scarcity of high-quality multimodal training samples

Innovation

• Mixture-of-experts framework with innovative gating function
• Handles missing modalities and irregularly sampled data
• Enhances convergence rates for better downstream performance