🤖 AI Summary
This work addresses the absence of a systematic privacy auditing framework for large language models (LLMs), which hinders the quantification of trade-offs between privacy and utility. The study extends the classification error gap (x-CEG) framework to the LLM setting, introducing LLM-CEG alongside an accompanying engineering pipeline, LLM-SIED. This approach integrates differentially private stochastic gradient descent (DP-SGD) fine-tuning, membership inference attacks to assess privacy leakage, and perplexity to evaluate utility, thereby enabling auditable and regulation-aligned privacy-compliant deployment. Experiments on DistilGPT-2 with synthetic clinical personally identifiable information (PII) demonstrate that the proposed method reduces attack success rates by 71.5% while improving out-of-distribution utility by 47–50%, confirming the implicit regularization effect of differential privacy under specific fine-tuning conditions.
📝 Abstract
This paper extends the Classification Error Gauge (x-CEG) framework, originally developed for measuring the privacy-utility trade-off in tabular datasets, to privacy auditing of Large Language Models (LLMs). We propose LLM-CEG, a systematic framework that employs membership inference attack (MIA) success rates as an empirical privacy gauge and model perplexity as a utility gauge, iteratively adjusting differential privacy parameters until both thresholds are jointly satisfied. A proof-of-concept prototype fine-tunes DistilGPT-2 on a synthetic clinical PII dataset under four privacy regimes using DP-SGD. Results indicate that DP-SGD reduces MIA attacker advantage by 71.5% while simultaneously improving out-of-distribution utility by 47-50% relative to the overfitted baseline, suggesting that differential privacy may act as implicit regularization under narrow fine-tuning conditions. We further extend the SIED engineering framework to the LLM context as LLM-SIED, providing an auditable, regulator-aligned process for privacy-compliant LLM deployment.