Communication-Efficient Wireless Federated Fine-Tuning for Large-Scale AI Models

2025-05-01

AI Summary

To address the high communication overhead and stringent resource constraints of fine-tuning large models in wireless federated learning, this paper proposes a LoRA-driven efficient collaborative training framework. Methodologically, it establishes a theoretical link between LoRA rank and data covariance for convergence analysis; introduces SOFT—a SVD-free, matrix-multiplication-free adaptive sparsification method; and develops TSFA, a two-stage algorithm integrating offline presetting with online dynamic tuning. The framework further incorporates orthogonal parameter updates, bandwidth-aware client scheduling, and channel-constrained modeling. Experimental results on benchmark datasets demonstrate that the proposed approach achieves accuracy comparable to full-parameter fine-tuning while reducing communication volume by over 60%. This significantly enhances scalability and real-time performance of large-model fine-tuning on resource-constrained edge devices.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

Transformer-based large language models (LLMs) have achieved remarkable success across various tasks. Yet, fine-tuning such massive models in federated learning (FL) settings poses significant challenges due to resource constraints and communication overhead. Low-Rank Adaptation (LoRA) addresses these issues by training compact, low-rank matrices instead of fully fine-tuning large models. This paper introduces a wireless federated LoRA fine-tuning framework that optimizes both learning performance and communication efficiency. We provide a novel convergence analysis, revealing how LoRA rank and covariance effects influence FL training dynamics. Leveraging these insights, we propose Sparsified Orthogonal Fine-Tuning ( extbf{SOFT}), an adaptive sparsification method that streamlines parameter updates without expensive matrix multiplications and singular value decomposition (SVD) operations. Additionally, we present a Two Stage Federated Algorithm ( extbf{TSFA}) algorithm that pre-determines key parameters offline and dynamically adjusts bandwidth and sparsification online, ensuring efficient training under latency constraints. Experiments on benchmark datasets show that our approach achieves accuracy comparable to ideal scenario models while significantly reducing communication overhead. Our framework thus enables scalable, resource-efficient deployment of large models in real-world wireless FL scenarios.

Problem

• Optimizes wireless federated fine-tuning for large AI models
• Reduces communication overhead in federated learning settings
• Ensures efficient training under resource and latency constraints

Innovation

• Wireless federated LoRA fine-tuning framework
• Sparsified Orthogonal Fine-Tuning (SOFT) method
• Two Stage Federated Algorithm (TSFA)