Multimodal Online Federated Learning with Modality Missing in Internet of Things

2025-05-22

AI Summary

To address the challenges of heterogeneity, dynamism, storage constraints, and frequent multimodal data missingness in IoT edge devices, this paper proposes a Multimodal Online Federated Learning (MMO-FL) framework. It is the first to integrate online learning, federated learning, and multimodal fault tolerance, introducing a Prototype-driven Modality Missingness Mitigation (PMM) algorithm that theoretically characterizes the impact of modality missingness on convergence. By jointly optimizing online gradient updates, federated aggregation, and prototype-based learning, MMO-FL achieves robust performance under stochastic modality dropout: retaining over 92% of full-modality accuracy and improving convergence stability by 40% on two real-world multimodal IoT datasets—outperforming existing baselines significantly. The core contribution is the establishment of the first online federated multimodal learning paradigm explicitly designed for dynamic, tolerance-aware modality missingness, backed by rigorous convergence analysis and theoretical guarantees.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

The Internet of Things (IoT) ecosystem generates vast amounts of multimodal data from heterogeneous sources such as sensors, cameras, and microphones. As edge intelligence continues to evolve, IoT devices have progressed from simple data collection units to nodes capable of executing complex computational tasks. This evolution necessitates the adoption of distributed learning strategies to effectively handle multimodal data in an IoT environment. Furthermore, the real-time nature of data collection and limited local storage on edge devices in IoT call for an online learning paradigm. To address these challenges, we introduce the concept of Multimodal Online Federated Learning (MMO-FL), a novel framework designed for dynamic and decentralized multimodal learning in IoT environments. Building on this framework, we further account for the inherent instability of edge devices, which frequently results in missing modalities during the learning process. We conduct a comprehensive theoretical analysis under both complete and missing modality scenarios, providing insights into the performance degradation caused by missing modalities. To mitigate the impact of modality missing, we propose the Prototypical Modality Mitigation (PMM) algorithm, which leverages prototype learning to effectively compensate for missing modalities. Experimental results on two multimodal datasets further demonstrate the superior performance of PMM compared to benchmarks.

Problem

• Handles multimodal data in IoT with missing modalities
• Proposes online federated learning for dynamic IoT environments
• Mitigates performance loss from missing modalities using prototypes

Innovation

• Multimodal Online Federated Learning for IoT
• Prototypical Modality Mitigation algorithm
• Handles missing modalities in decentralized learning