FedSmoothLoRA: Toward Smoother and Faster Convergence in Federated Low-Rank Adaptation

2026-05-28

AI Summary

Federated LoRA fine-tuning faces three key challenges: constrained update space, inter-round state misalignment, and client-agnostic initialization, which collectively lead to slow convergence and discontinuous training dynamics. This work proposes FedSmoothLoRA, the first framework to simultaneously address inter-round state misalignment and client-agnostic initialization. It introduces a round-matching matrix to preserve local optimization continuity across communication rounds and a gradient-alignment matrix to enable client-aware initialization. While retaining the low-rank adaptation capability of LoRA, FedSmoothLoRA substantially enhances training smoothness and convergence speed. Experimental results demonstrate that FedSmoothLoRA consistently outperforms existing federated LoRA methods on both image classification and natural language generation tasks, achieving faster convergence and superior final performance.

Abstract

Federated fine-tuning of foundation models with Low-Rank Adaptation (LoRA) provides an efficient solution for reducing communication and computation costs while preserving data locality. However, the direct combination of FedAvg and LoRA suffers from three key issues: limited update space, which restricts the model's effective learning capacity; inter-round state mismatch, which disrupts cross-round local optimization continuity; and a client-agnostic starting state, which slows local convergence on clients. Although recent methods mitigate the limited update space issue by merging LoRA updates into the backbone across communication rounds, inter-round state mismatch and the client-agnostic starting state remain insufficiently addressed. To address these issues, we propose FedSmoothLoRA, a federated LoRA tuning framework that preserves the enlarged update space, improves cross-round local optimization continuity, and provides a client-aware starting state for local training. At each communication round, FedSmoothLoRA constructs the local LoRA initialization using two matrices: a Round-Matching matrix that preserves cross-round local state continuity, and a Gradient-Aligned matrix that provides client-specific optimization guidance from gradient signals estimated on local data. Together, these designs enable smoother and faster convergence. Extensive experiments on image classification and natural language generation tasks demonstrate that FedSmoothLoRA consistently outperforms existing federated LoRA tuning methods. Code: https://github.com/wangzehao0704/FedSmoothLoRA

Problem

• Federated Learning
• Low-Rank Adaptation
• Convergence
• State Mismatch
• Client-Aware Initialization

Innovation

• Federated Learning
• Low-Rank Adaptation
• Convergence Acceleration
• Client-Aware Initialization
• State Continuity