PPC-GPT: Federated Task-Specific Compression of Large Language Models via Pruning and Chain-of-Thought Distillation

2025-02-21

AI Summary

To address the dual challenges of domain-knowledge privacy preservation and model lightweighting for LLM deployment in resource-constrained settings, this paper proposes a federated task-customized compression framework. The framework introduces a novel synergistic paradigm integrating differential privacy (ε = 2.0), LLM-driven chain-of-thought (CoT) synthetic data generation, structured pruning, and CoT-aware knowledge distillation: privacy-sensitive data perturbation and synthesis occur locally on clients, while efficient compression is performed centrally at the server. Evaluated across multiple text generation tasks, the resulting small language model (SLM) achieves over 92% of the original LLM’s performance while reducing communication overhead by 5.3×—significantly outperforming existing federated compression approaches. Our core contribution is the first end-to-end federated compression paradigm that simultaneously ensures rigorous privacy guarantees, high-fidelity domain-knowledge retention, and computational efficiency.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Data Mining & Knowledge Management: Data Compression
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Compressing Large Language Models (LLMs) into task-specific Small Language Models (SLMs) encounters two significant challenges: safeguarding domain-specific knowledge privacy and managing limited resources. To tackle these challenges, we propose PPC-GPT, a innovative privacy-preserving federated framework specifically designed for compressing LLMs into task-specific SLMs via pruning and Chain-of-Thought (COT) distillation. PPC-GPT works on a server-client federated architecture, where the client sends differentially private (DP) perturbed task-specific data to the server's LLM. The LLM then generates synthetic data along with their corresponding rationales. This synthetic data is subsequently used for both LLM pruning and retraining processes. Additionally, we harness COT knowledge distillation, leveraging the synthetic data to further improve the retraining of structurally-pruned SLMs. Our experimental results demonstrate the effectiveness of PPC-GPT across various text generation tasks. By compressing LLMs into task-specific SLMs, PPC-GPT not only achieves competitive performance but also prioritizes data privacy protection.

Problem

• Compress LLMs into task-specific SLMs.
• Protect domain-specific knowledge privacy.
• Manage limited computational resources effectively.

Innovation

• Federated task-specific compression
• Pruning and Chain-of-Thought distillation
• Differentially private data perturbation