FedProxy: Federated Fine-Tuning of LLMs via Proxy SLMs and Heterogeneity-Aware Fusion

2026-04-20

AI Summary

This work addresses the trilemma in federated large language model (LLM) fine-tuning—balancing model intellectual property protection, client-side privacy preservation, and performance degradation under heterogeneous data distributions. To this end, we propose FedProxy, a novel framework that introduces, for the first time, a unified proxy small model distilled from a proprietary LLM to replace conventional lightweight adapters, enabling efficient collaborative fine-tuning in federated settings. FedProxy integrates server-guided proxy construction, heterogeneity-aware aggregation, and a training-agnostic knowledge reinjection mechanism to systematically overcome these challenges. Experimental results demonstrate that FedProxy substantially outperforms existing offsite-tuning approaches and achieves performance nearly on par with centralized training, establishing a new benchmark for secure and efficient federated adaptation of large models.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Federated fine-tuning of Large Language Models (LLMs) is obstructed by a trilemma of challenges: protecting LLMs intellectual property (IP), ensuring client privacy, and mitigating performance loss on heterogeneous data. Existing methods like Offsite-Tuning (OT) secure the LLMs IP by having clients train only lightweight adapters, yet our analysis reveals they suffer from a fundamental performance bottleneck, leaving a significant gap compared to centralized training. To bridge this gap, we introduce FedProxy, a new federated adaptation framework. FedProxy replaces weak adapters with a unified, powerful Proxy Small Language Model (SLM), compressed from the proprietary LLM, to serve as a high-fidelity surrogate for collaborative fine-tuning. Our framework systematically resolves the trilemma through a three-stage architecture: (i) Efficient Representation via server-guided compression to create a resource-friendly proxy; (ii) Robust Optimization through an interference-mitigating aggregation strategy to handle data heterogeneity; and (iii) Effortless Fusion via a training-free "plug-in" mechanism to integrate learned knowledge back into the LLM. Experiments show FedProxy significantly outperforms OT methods and approaches centralized performance, establishing a new benchmark for secure and high-performance federated LLM adaptation.

Problem

• Federated Learning
• Large Language Models
• Intellectual Property
• Data Heterogeneity
• Privacy Preservation

Innovation

• Federated Learning
• Large Language Models
• Proxy Small Language Model
• Heterogeneity-Aware Fusion
• Intellectual Property Protection