PRvL: Quantifying the Capabilities and Risks of Large Language Models for PII Redaction

2025-08-07

AI Summary

This paper addresses the challenge of quantifying performance and privacy risks of large language models (LLMs) in personally identifiable information (PII) anonymization. To this end, we propose a systematic evaluation framework and introduce PRvL—an open-source toolkit built upon mainstream open-weight LLMs. Our approach designs a multidimensional evaluation protocol grounded in named entity recognition (NER), systematically analyzing how model architecture, fine-tuning strategies, and inference configurations affect anonymization accuracy, semantic preservation, and privacy leakage risk. Key contributions include: (1) the first reproducible, decoupled quantitative analysis of PII anonymization capability and associated privacy risks; (2) support for local deployment and adaptation across diverse application scenarios; and (3) an end-to-end solution that achieves high anonymization accuracy (F1 > 92%), while significantly reducing semantic distortion and inference latency—thereby balancing regulatory compliance, practical utility, and customization flexibility.

Technology Category

• Machine Learning: Privacy
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Redacting Personally Identifiable Information (PII) from unstructured text is critical for ensuring data privacy in regulated domains. While earlier approaches have relied on rule-based systems and domain-specific Named Entity Recognition (NER) models, these methods fail to generalize across formats and contexts. Recent advances in Large Language Models (LLMs) offer a promising alternative, yet the effect of architectural and training choices on redaction performance remains underexplored. LLMs have demonstrated strong performance in tasks that require contextual language understanding, including the redaction of PII in free-form text. Prior work suggests that with appropriate adaptation, LLMs can become effective contextual privacy learners. However, the consequences of architectural and training choices for PII Redaction remain underexplored. In this work, we present a comprehensive analysis of LLMs as privacy-preserving PII Redaction systems. We evaluate a range of LLM architectures and training strategies for their effectiveness in PII Redaction. Our analysis measures redaction performance, semantic preservation, and PII leakage, and compares these outcomes against latency and computational cost. The results provide practical guidance for configuring LLM-based redactors that are accurate, efficient, and privacy-aware. To support reproducibility and real-world deployment, we release PRvL, an open-source suite of fine-tuned models, and evaluation tools for general-purpose PII Redaction. PRvL is built entirely on open-source LLMs and supports multiple inference settings for flexibility and compliance. It is designed to be easily customized for different domains and fully operable within secure, self-managed environments. This enables data owners to perform redactions without relying on third-party services or exposing sensitive content beyond their own infrastructure.

Problem

• Evaluating LLMs for effective PII redaction in text
• Assessing architectural and training impacts on privacy performance
• Developing open-source tools for customizable, secure PII removal

Innovation

• Uses LLMs for PII redaction in text
• Evaluates architectures and training strategies
• Provides open-source tools for customization