PFAdapter: Hierarchical LoRA Decomposition for Personalized Federated MLLMs

2026-07-13

AI Summary

This work addresses key challenges in edge-based federated fine-tuning of multimodal large language models, including the trade-off between global knowledge aggregation and local personalization, high communication overhead, and parameter interference. To overcome these issues, the authors propose PFAdapter, a novel framework featuring a hierarchical LoRA architecture that explicitly decomposes adapter parameters into globally shared and locally private components. This decomposition is rigorously enforced through Frobenius norm–based orthogonal regularization, effectively disentangling universal semantics from local features. A selective aggregation strategy further reduces communication costs by synchronizing only the global components in query and key projections, while keeping value and output projections updated locally. Experiments demonstrate that PFAdapter achieves accuracy gains of 2.4%–4.8% on benchmarks such as VQA-RAD and SLAKE, while cutting communication overhead by nearly 50%, substantially outperforming existing baselines.

Abstract

Agentic AI systems are reshaping communications and networking by deploying autonomous intelligent agents capable of collaborative learning while maintaining data privacy at network edges. Within distributed network environments, Multimodal Large Language Models (MLLMs) serve as cognitive engines for edge devices, yet federated fine-tuning faces substantial challenges in balancing global knowledge aggregation with local adaptation under heterogeneous network conditions. Conventional federated protocols typically rely on uniform parameter aggregation, which conflates domain-invariant features with client-specific nuances, thereby resulting in suboptimal personalization and excessive communication overhead. To address these challenges, we propose PFAdapter, a communication-efficient framework introducing hierarchical LoRA decomposition to explicitly separate adapter parameters into global-shared and local-private components. Query and key projections are assigned to global synchronization for capturing universal multimodal semantics across the network, while value and output projections remain localized for edge-specific adaptation. Additionally, orthogonality regularization based on the Frobenius norm enforces strict separation between these components, preventing redundant feature learning. Selective aggregation protocols synchronize only global-shared components across the federated network, preserving local expertise and reducing communication costs by nearly 50%. Extensive experiments on VQA-RAD, SLAKE, Hateful Memes, and CrisisMMD datasets demonstrate that PFAdapter consistently outperforms state-of-the-art baselines, achieving accuracy improvements ranging from 2.4% to 4.8% across diverse edge intelligence tasks. Consequently, our framework establishes an efficient solution for agentic AI deployment in resource-constrained communication networks.

Problem

• Personalized Federated Learning
• Multimodal Large Language Models
• Communication Efficiency
• Parameter Heterogeneity
• Edge Intelligence

Innovation

• Hierarchical LoRA Decomposition
• Personalized Federated Learning
• Multimodal Large Language Models
• Communication Efficiency
• Orthogonality Regularization