SoK: The Privacy Paradox of Large Language Models: Advancements, Privacy Risks, and Mitigation

2025-06-15

AI Summary

This work addresses previously overlooked privacy risks in large language models (LLMs) arising from user interactions and agent-level behaviors. We propose the first four-dimensional privacy leakage taxonomy—encompassing training data, user prompts, generated outputs, and agent actions—to systematically uncover novel cross-layer leakage pathways. Through a systematic literature review (SoK), multi-layer privacy impact analysis, and comparative evaluation of mitigation techniques, we identify critical limitations of conventional de-identification and differential privacy methods in LLM contexts. Our study bridges a key gap in privacy research at the user interaction and autonomous agent layers, establishes a practical, deployment-oriented framework for evaluating privacy mitigation efficacy, and explicitly identifies six fundamental research gaps. The findings provide both theoretical foundations and actionable guidelines for designing privacy-enhancing LLMs.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Security and Privacy: Large-scale security measurements

Abstract

Large language models (LLMs) are sophisticated artificial intelligence systems that enable machines to generate human-like text with remarkable precision. While LLMs offer significant technological progress, their development using vast amounts of user data scraped from the web and collected from extensive user interactions poses risks of sensitive information leakage. Most existing surveys focus on the privacy implications of the training data but tend to overlook privacy risks from user interactions and advanced LLM capabilities. This paper aims to fill that gap by providing a comprehensive analysis of privacy in LLMs, categorizing the challenges into four main areas: (i) privacy issues in LLM training data, (ii) privacy challenges associated with user prompts, (iii) privacy vulnerabilities in LLM-generated outputs, and (iv) privacy challenges involving LLM agents. We evaluate the effectiveness and limitations of existing mitigation mechanisms targeting these proposed privacy challenges and identify areas for further research.

Problem

• Analyzing privacy risks in LLM training data and user interactions
• Identifying privacy vulnerabilities in LLM-generated outputs and agents
• Evaluating mitigation mechanisms for LLM privacy challenges

Innovation

• Analyzes privacy risks in LLM training data
• Examines privacy challenges from user prompts
• Assesses vulnerabilities in LLM-generated outputs