SeFoRA: Sketch-Aggregated Federated Low-Rank Adaptation with Heterogeneous Client Ranks

2026-08-10

AI Summary

This work addresses the challenges of incompatible factor matrix dimensions and bilinear mismatch arising from heterogeneous rank configurations in federated low-rank adaptation (LoRA). To overcome these issues, the authors propose SeFoRA, a novel algorithm that introduces, for the first time, a linear sketch–based aggregation mechanism. This allows clients to upload low-dimensional sketches of their local LoRA updates, enabling the server to directly aggregate them within a shared subspace and thereby circumvent bilinear mismatch. The paper also presents a homogeneous variant, SeFoRA-Ho, and provides theoretical analysis establishing its convergence rate at 𝑂(1/𝑇). Experimental results demonstrate that SeFoRA significantly outperforms existing federated parameter-efficient fine-tuning methods when fine-tuning RoBERTa-Large on the GLUE benchmark.

Abstract

We consider federated parameter efficient fine-tuning of large neural networks with low-rank adaptation (LoRA,~Hu et al.\ 2022). Combining LoRA with federated PEFT introduces challenges absent from either setting alone: clients may use different LoRA ranks, making their factor matrices dimension-incompatible, and factor-wise averaging suffers from a bilinear mismatch. We propose SeFoRA, a sketch-aggregated federated LoRA algorithm in which each client transmits a linear sketch of its local updates, enabling direct aggregation at the federator. As a result, SeFoRA alleviates the bilinear mismatch, and allows for aggregation in a small subspace of the full model. We introduce a rank-homogeneous version called SeFoRA-Ho which allows for direct adapter aggregation in this setting. We prove convergence to a neighborhood of the first-order stationary point at rate $\cO(1/T)$ for the rank-homogeneous setting. Numerical experiments on fine-tuning RoBERTa-Large on GLUE datasets show how our algorithms outperform the state-of-the-art.

Problem

• Federated Learning
• Low-Rank Adaptation
• Parameter-Efficient Fine-Tuning
• Heterogeneous Client Ranks
• Bilinear Mismatch

Innovation

• Federated Learning
• Low-Rank Adaptation
• Parameter-Efficient Fine-Tuning
• Linear Sketching
• Heterogeneous Clients