The Art of Mixology: Mixup-based Obfuscation for Privacy-Preserving Split Learning in Large Language Models

2026-06-15

AI Summary

This work addresses the challenge of simultaneously achieving utility, privacy, efficiency, and robustness in split learning for large language models, where existing privacy-preserving approaches often suffer from vulnerability to advanced data reconstruction attacks and unstable performance. To overcome these limitations, we propose MIXGUARD, a novel framework that introduces mixup-inspired techniques into split learning for the first time. MIXGUARD integrates token-level and representation-level obfuscation with adaptive gradient perturbation to preserve informative learning signals while safeguarding user privacy. A lightweight calibration model trained on public data enables an efficient and stable multi-level obfuscation mechanism. Extensive experiments across eight tasks demonstrate that MIXGUARD matches the utility of non-split training while significantly outperforming state-of-the-art methods in resisting sophisticated reconstruction attacks and maintaining robustness under adaptive adversarial settings.

Abstract

Split learning provides a practical paradigm for resource-constrained users to train Large Language Models (LLMs) by offloading computation-intensive layers to a server while keeping raw data local. However, existing privacy-preserving split learning methods still face a difficult trade-off among utility, privacy, efficiency, and stability. Specifically, these methods often suffer from substantial utility degradation, remain vulnerable to advanced data reconstruction attacks, incur prohibitive computational and communication overhead, or exhibit unstable performance across different tasks. In this paper, we propose MIXGUARD, a novel mixup-based privacy-preserving split learning framework for LLMs. MIXGUARD introduces token-level obfuscation, representation-level obfuscation, and adaptive gradient perturbation mechanisms, which operate jointly to preserve useful learning signals while preventing privacy leakage to the server. Technically, MIXGUARD first constructs a lightweight calibration model on a public dataset to refine the approximated target representation, and then applies this model during privacy-preserving fine-tuning on private data. We conduct extensive experiments on four classification tasks and four text generation tasks across multiple LLM families, model sizes, architectures, and fine-tuning strategies. The results show that MIXGUARD preserves model utility comparable to non-split training baselines, consistently achieves stronger privacy protection than existing split learning defense methods against state-of-the-art data reconstruction attacks, and remains robust under adaptive attack settings.

Problem

• split learning
• privacy-preserving
• large language models
• data reconstruction attacks
• utility-privacy trade-off

Innovation

• mixup-based obfuscation
• split learning
• privacy-preserving LLMs
• adaptive gradient perturbation
• token-level obfuscation