FLoKD: Adaptive Knowledge Distillation for Federated Low-Rank LLM over Wireless Networks

2026-09-11

AI Summary

为解决联邦学习中大模型参数传输的通信开销问题，提出FLoKD框架，通过传递中间激活而非logits或全参数，并选择性传输重要块来减少通信成本。

Abstract

Large language models (LLMs) have demonstrated strong capabilities across a wide range of natural language processing tasks. However, conventional fine-tuning typically relies on centralized data collection, bringing in privacy concerns. Federated learning (FL) enables collaborative LLM fine-tuning without sharing raw client data, but its deployment over bandwidth-constrained wireless networks is hindered by the communication overhead of model-parameter transmission. Although Low-Rank Adaptation (LoRA) reduces the number of trainable parameters, its communication cost still increases with model scale. Knowledge distillation avoids parameter sharing via output logits, but token-level logits in LLMs incur high communication cost due to sequence length and vocabulary size. Reducing logits lowers the cost but weakens supervision and degrades accuracy. To address these limitations, we propose FLoKD, an adaptive knowledge-distillation framework for federated LoRA fine-tuning of LLMs over wireless networks, which communicates intermediate LoRA activations as the distillation signal rather than logits or full parameters. Since transmitting all blocks over the entire public dataset remains costly, we further propose a transformer block importance scoring framework that selectively transmits the most informative blocks, and two dataset selection strategies that discard public samples deviating from the local data distribution and prioritise those most informative for distillation. Extensive experiments across multiple generative language datasets, including WikiText-103, PTB, and Dialog, demonstrate that our proposed framework reduces communication overhead by 50-65% while achieving rapid convergence to competitive perplexity compared to baselines.

Problem

• Federated Learning
• Wireless Networks
• Large Language Models
• Communication Overhead
• Knowledge Distillation

Innovation

• adaptive knowledge distillation
• federated learning
• low-rank adaptation
• communication overhead reduction
• transformer block importance scoring