PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Action

2024-08-29

Neural Information Processing Systems

AI Summary

This work addresses two key challenges in evaluating large language models (LLMs) for privacy-aware personalized communication: the difficulty of quantifying privacy norm awareness and insufficient assessment coverage of long-tail sensitive scenarios. To this end, we propose PrivacyLens—a novel framework that formalizes privacy norms as scalable agent behavior trajectories. PrivacyLens constructs a normative knowledge base by synthesizing privacy literature and crowdsourced data, then generates high-fidelity vignettes and corresponding behavioral trajectories via seed expansion, context augmentation, and multi-round agent simulation. It further supports dynamic red-teaming. Experimental evaluation reveals a substantial gap between LLMs’ question-answering capability and their actual privacy-preserving behavior: GPT-4 and Llama-3-70B leak sensitive information in 25.68% and 38.69% of privacy-sensitive cases, respectively, even under agent-mode prompting. We publicly release our dataset and code to advance standardized privacy evaluation for language models.

Technology Category

• Machine Learning: Privacy
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Philosophy and Ethics of AI: Privacy & Security

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Security and Privacy: Large-scale security measurements

Abstract

As language models (LMs) are widely utilized in personalized communication scenarios (e.g., sending emails, writing social media posts) and endowed with a certain level of agency, ensuring they act in accordance with the contextual privacy norms becomes increasingly critical. However, quantifying the privacy norm awareness of LMs and the emerging privacy risk in LM-mediated communication is challenging due to (1) the contextual and long-tailed nature of privacy-sensitive cases, and (2) the lack of evaluation approaches that capture realistic application scenarios. To address these challenges, we propose PrivacyLens, a novel framework designed to extend privacy-sensitive seeds into expressive vignettes and further into agent trajectories, enabling multi-level evaluation of privacy leakage in LM agents' actions. We instantiate PrivacyLens with a collection of privacy norms grounded in privacy literature and crowdsourced seeds. Using this dataset, we reveal a discrepancy between LM performance in answering probing questions and their actual behavior when executing user instructions in an agent setup. State-of-the-art LMs, like GPT-4 and Llama-3-70B, leak sensitive information in 25.68% and 38.69% of cases, even when prompted with privacy-enhancing instructions. We also demonstrate the dynamic nature of PrivacyLens by extending each seed into multiple trajectories to red-team LM privacy leakage risk. Dataset and code are available at https://github.com/SALT-NLP/PrivacyLens.

Problem

• Evaluating privacy norm awareness in language models.
• Quantifying privacy risks in LM-mediated communication.
• Developing a framework to assess LM privacy leakage.

Innovation

• PrivacyLens framework evaluates LM privacy norms.
• Extends seeds into vignettes and agent trajectories.
• Red-teams LM privacy leakage risk dynamically.