MAESTRO : Adaptive Sparse Attention and Robust Learning for Multimodal Dynamic Time Series

2025-09-28

AI Summary

Existing multimodal time-series modeling approaches suffer from three key limitations: reliance on a single dominant modality for alignment, restriction to pairwise modality interactions, and the assumption of complete modality observations. To address these, we propose an adaptive sparse attention and robust learning framework that employs symbolic tokenization, dynamic attention budget allocation, sparse cross-modal attention, and a sparse Mixture-of-Experts mechanism. This enables task-driven, black-box multimodal composition modeling and inherently supports dynamic inter-modality interaction under arbitrary modality missingness. Crucially, the framework eliminates the need for dominant-modality priors and pairwise interaction constraints, substantially enhancing generalizability and robustness. Evaluated on four benchmark datasets against ten baselines, our method achieves average improvements of 4% (multimodal) and 8% (multivariate) under full observation, and maintains a 9% average gain even with 40% randomly missing modalities.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

From clinical healthcare to daily living, continuous sensor monitoring across multiple modalities has shown great promise for real-world intelligent decision-making but also faces various challenges. In this work, we introduce MAESTRO, a novel framework that overcomes key limitations of existing multimodal learning approaches: (1) reliance on a single primary modality for alignment, (2) pairwise modeling of modalities, and (3) assumption of complete modality observations. These limitations hinder the applicability of these approaches in real-world multimodal time-series settings, where primary modality priors are often unclear, the number of modalities can be large (making pairwise modeling impractical), and sensor failures often result in arbitrary missing observations. At its core, MAESTRO facilitates dynamic intra- and cross-modal interactions based on task relevance, and leverages symbolic tokenization and adaptive attention budgeting to construct long multimodal sequences, which are processed via sparse cross-modal attention. The resulting cross-modal tokens are routed through a sparse Mixture-of-Experts (MoE) mechanism, enabling black-box specialization under varying modality combinations. We evaluate MAESTRO against 10 baselines on four diverse datasets spanning three applications, and observe average relative improvements of 4% and 8% over the best existing multimodal and multivariate approaches, respectively, under complete observations. Under partial observations -- with up to 40% of missing modalities -- MAESTRO achieves an average 9% improvement. Further analysis also demonstrates the robustness and efficiency of MAESTRO's sparse, modality-aware design for learning from dynamic time series.

Problem

• Overcoming reliance on single primary modality for multimodal alignment
• Addressing impractical pairwise modeling with large modality numbers
• Handling arbitrary missing observations due to sensor failures

Innovation

• Dynamic intra- and cross-modal interaction mechanism
• Symbolic tokenization with adaptive attention budgeting
• Sparse Mixture-of-Experts routing for modality combinations