Federated LoRA Fine-Tuning for LLMs via Collaborative Alignment

2026-05-20

AI Summary

This work addresses the challenge of efficiently and robustly fine-tuning low-rank adaptation (LoRA) modules of large language models in highly heterogeneous federated learning settings with a substantial fraction of poisoned clients. The proposed CLAIR framework introduces, for the first time, a contamination-aware mechanism into federated LoRA fine-tuning. Relying solely on local preliminary estimates, CLAIR jointly recovers a shared LoRA subspace and identifies malicious clients through structured low-rank plus block-sparse decomposition. Notably, it achieves cross-client subspace alignment without requiring global communication and comes with theoretical guarantees of stable recovery under noise and consensus among benign clients. Experiments demonstrate that CLAIR accurately detects poisoned clients in text replication tasks and significantly outperforms both local fine-tuning and non-robust federated averaging in terms of accuracy and robustness for benign participants.

Abstract

Low-rank adaptation (LoRA) has emerged as a powerful tool for parameter-efficient fine-tuning of large language models (LLMs). This paper studies LoRA under a federated learning setting, enabling collaborative fine-tuning across clients while preserving parameter efficiency. We focus on a highly heterogeneous regime in which clients share only partial structure and a substantial subset may be contaminated. We propose Collaborative Low-rank Alignment and Identifiable Recovery (CLAIR), a contamination-aware framework that relies only on preliminary local estimators. Its formulation applies broadly, from linear regression to neural network and LLM modules, whenever local adaptation can be represented by matrix-valued updates. CLAIR recovers the shared LoRA subspace and detects contaminated clients via a structured low-rank plus block-sparse decomposition. We prove exact recovery of the shared LoRA subspace in the noiseless case, stable recovery under preliminary estimation error, and consistent collaborative-set recovery under mild separation conditions. We further quantify the gain from CLAIR refinement: it reduces off-subspace estimation error through cross-client averaging while preserving client-specific variation within the shared LoRA subspace, thus improves over local fine-tuning whenever this oracle gain outweighs the costs of subspace estimation and benign-client heterogeneity. Empirically, we demonstrate the benefits of CLAIR by fine-tuning a Transformer architecture on a text-copying task. The results show accurate contamination detection and improved benign-client performance compared with local fine-tuning and non-robust federated averaging.

Problem

• Federated Learning
• LoRA
• Large Language Models
• Client Heterogeneity
• Contamination

Innovation

• Federated Learning
• LoRA
• Low-rank Adaptation
• Contamination Robustness
• Structured Matrix Decomposition