Privacy Enhanced PEFT: Tensor Train Decomposition Improves Privacy Utility Tradeoffs under DP-SGD

2026-01-15

AI Summary

This work addresses the vulnerability of large language models fine-tuned on sensitive data to membership inference attacks, a risk exacerbated when combining existing parameter-efficient methods like LoRA with differential privacy (DP), often resulting in significant utility degradation. To mitigate this trade-off, the authors propose TTLoRA-DP, a novel framework that integrates Tensor Train decomposition into DP-aware fine-tuning. This approach drastically reduces the number of trainable parameters—by an average factor of 7.6—while preserving model expressiveness. Furthermore, TTLoRA-DP introduces a cache-shrinking mechanism that enables precise gradient clipping without explicitly computing full gradients, thereby enhancing the efficiency of DP-SGD. Experiments on GPT-2 using the Enron and Penn Treebank datasets demonstrate that TTLoRA-DP achieves comparable or superior utility relative to LoRA-DP, offers stronger privacy guarantees, and exhibits lower membership leakage even in non-DP settings.

Technology Category

• Machine Learning: Privacy
• Natural Language Processing: Safety and Robustness
• Computer Vision: Diffusion Models for Vision

Application Category

• Security and Privacy: Privacy-enhancing technologies
• Responsible Web: Data and user privacy-enhancing technologies for the Web
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems

Abstract

Fine-tuning large language models on sensitive data poses significant privacy risks, as membership inference attacks can reveal whether individual records were used during training. While Differential Privacy (DP) provides formal protection, applying DP to conventional Parameter-Efficient Fine-Tuning (PEFT) methods such as Low-Rank Adaptation (LoRA) often incurs substantial utility loss. In this work, we show that a more structurally constrained PEFT architecture, Tensor Train Low-Rank Adaptation (TTLoRA), can improve the privacy-utility tradeoff by shrinking the effective parameter space while preserving expressivity. To this end, we develop TTLoRA-DP, a differentially private training framework for TTLoRA. Specifically, we extend the ghost clipping algorithm to Tensor Train cores via cached contraction states, enabling efficient Differentially Private Stochastic Gradient Descent (DP-SGD) with exact per-example gradient norm computation without materializing full per-example gradients. Experiments on GPT-2 fine-tuning over the Enron and Penn Treebank datasets show that TTLoRA-DP consistently strengthens privacy protection relative to LoRA-DP while maintaining comparable or better downstream utility. Moreover, TTLoRA exhibits lower membership leakage even without DP training, using substantially smaller adapters and requiring on average 7.6X fewer parameters than LoRA. Overall, our results demonstrate that TTLoRA offers a practical path to improving the privacy-utility tradeoff in parameter-efficient language model adaptation.

Problem

• Differential Privacy
• Parameter-Efficient Fine-Tuning
• Privacy-Utility Tradeoff
• Membership Inference Attack
• Large Language Models

Innovation

• Tensor Train Decomposition
• Differential Privacy
• Parameter-Efficient Fine-Tuning
• DP-SGD
• TTLoRA