Communication-Efficient Personalized Adaptation via Federated-Local Model Merging

2026-02-20

AI Summary

This work addresses the challenge in federated learning where task heterogeneity makes it difficult to balance model generalization and personalization. To this end, the authors propose Potara, a novel framework that provides the first theoretical foundation for personalized model fusion in federated settings. Leveraging linear mode connectivity theory, Potara derives a closed-form solution for optimal mixing weights that efficiently combine a global model with locally fine-tuned models—such as those adapted via LoRA—yielding a merged model whose loss upper bound is provably tighter than either constituent model alone. Extensive experiments demonstrate that Potara significantly enhances personalized performance across vision and language benchmarks while substantially reducing communication overhead, achieving an excellent trade-off between accuracy and communication efficiency.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Search and Optimization: Learning to Search
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Parameter-efficient fine-tuning methods, such as LoRA, offer a practical way to adapt large vision and language models to client tasks. However, this becomes particularly challenging under task-level heterogeneity in federated deployments. In this regime, personalization requires balancing general knowledge with personalized knowledge, yet existing approaches largely rely on heuristic mixing rules and lack theoretical justification. Moreover, prior model merging approaches are also computation and communication intensive, making the process inefficient in federated settings. In this work, we propose Potara, a principled framework for federated personalization that constructs a personalized model for each client by merging two complementary models: (i) a federated model capturing general knowledge, and (ii) a local model capturing personalized knowledge. Through the construct of linear mode connectivity, we show that the expected task loss admits a variance trace upper bound, whose minimization yields closed-form optimal mixing weights that guarantee a tighter bound for the merged model than for either the federated or local model alone. Experiments on vision and language benchmarks show that Potara consistently improves personalization while reducing communication, leading to a strong performance-communication trade-off.

Problem

• federated learning
• personalization
• model merging
• task heterogeneity
• communication efficiency

Innovation

• federated learning
• personalization
• model merging
• parameter-efficient fine-tuning
• linear mode connectivity