Randomized Masked Finetuning: An Efficient Way to Mitigate Memorization of PIIs in LLMs

2025-12-02

AI Summary

Large language models (LLMs) often overfit and memorize personally identifiable information (PII) during training, posing severe privacy risks. To address this, we propose Randomized Masking Fine-Tuning (RMFT), a lightweight fine-tuning method that dynamically masks sensitive PII fields during adaptation while preserving model utility via parameter-efficient mechanisms. We further introduce MaxTER, a systematic evaluation framework, and the Area Under the Robustness–Capability curve (AURC) metric—the first to jointly and quantitatively characterize the trade-off between PII extraction rate and language modeling performance. Experiments on the Enron email dataset show RMFT reduces total PII extraction by 80.81% and seen-PII extraction by 80.17%, with only a 5.73% increase in perplexity—substantially outperforming baselines such as deduplication. This work establishes an efficient, rigorously evaluable, and low-overhead paradigm for privacy-preserving LLM training.

Technology Category

• Machine Learning: Privacy
• Natural Language Processing: Safety and Robustness
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Security and Privacy: Large-scale security measurements

Abstract

The current literature on memorization in Natural Language Models, especially Large Language Models (LLMs), poses severe security and privacy risks, as models tend to memorize personally identifying information (PIIs) from training data. We introduce Randomized Masked Fine-Tuning (RMFT), a novel privacy-preserving fine-tuning technique that reduces PII memorization while minimizing performance impact. Using the Enron Email Dataset, we demonstrate that RMFT achieves an 80.81% reduction in Total Extraction Rate and 80.17% reduction in Seen Extraction Rate compared to baseline fine-tuning, outperforming deduplication methods while maintaining only a 5.73% increase in perplexity. We present MaxTER, a Pareto-optimal evaluation framework for assessing privacy-utility tradeoffs, and show the performance of RMFT vs Deduplication by Area Under The Response Curve (AURC) metric.

Problem

• Mitigates LLMs' memorization of personal data from training
• Reduces privacy risks from PII exposure in language models
• Balances privacy protection with minimal performance degradation

Innovation

• Randomized Masked Fine-Tuning reduces PII memorization
• RMFT cuts extraction rates by over 80% versus baseline
• Method maintains utility with minimal perplexity increase