RELIEF: Turning Missing Modalities into Training Acceleration for Federated Learning on Heterogeneous IoT Edge

2026-04-05

AI Summary

This work addresses the challenges of low training efficiency and dilution of rare-modal information in heterogeneous edge federated learning for the Internet of Things, where system, modality, and data heterogeneities are tightly coupled. To tackle these issues, the authors propose a modality-aligned grouped aggregation mechanism that partitions LoRA projection matrices into column blocks according to modality, enabling devices to train and communicate in modality-specific groups. This design eliminates cross-modal gradient interference and dynamically allocates personalized training budgets based on intra-group divergence. The proposed method is the first to jointly handle all three forms of heterogeneity while providing theoretical guarantees of convergence and sublinear regret. Experiments on PAMAP2 and MHEALTH datasets demonstrate up to 9.41× faster convergence, 37% lower energy consumption, and a 15.3-percentage-point improvement in F1 score for rare modalities, with significant gains validated on Jetson edge devices.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Federated learning (FL) over heterogeneous IoT edge devices faces coupled system-modality-data heterogeneity: the lower-cost device carries both fewer sensors and less computational power, so the slowest device (straggler) produces the most incomplete gradient signals. Naively averaging their updates dilutes rare-modality information and wastes computation on absent-sensor parameters, whereas existing methods handle the triple heterogeneity (system, modality, data) in isolation and none addresses their coupling. To resolve this issue, we propose RELIEF, a framework that partitions the fusion-layer Low-Rank Adaptation (LoRA) projection matrix into modality-aligned column blocks and uses this partition as a unified interface for aggregation, elastic training, and communication. Each block is aggregated only within the cohort of devices possessing that modality, which eliminates cross-modal gradient interference; the server then allocates personalized training budgets by prioritizing blocks with the highest cohort-internal divergence, so that resource-constrained devices train fewer but more impactful parameters. We prove that cohort-wise aggregation removes interference from the convergence bound and that the divergence-guided allocation achieves sublinear regret. Experiments on two IoT sensor datasets (PAMAP2, MHEALTH) under both full-parameter (CNN) and parameter-efficient (LoRA) training show that RELIEF achieves up to 9.41x speedup and 37% energy reduction over FedAvg with up to 15.3 pp rare-modality F1 gains, and real-device validation on a two-Jetson AGX Orin testbed confirms these results.

Problem

• federated learning
• heterogeneous IoT edge
• modality heterogeneity
• system heterogeneity
• data heterogeneity

Innovation

• modality-aligned aggregation
• cohort-wise training
• divergence-guided budget allocation
• federated learning
• parameter-efficient fine-tuning