Differential Privacy in Generative AI Agents: Analysis and Optimal Tradeoffs

2026-03-18

AI Summary

This work addresses the risk of inadvertent sensitive information leakage when generative AI agents access internal enterprise databases, a concern inadequately mitigated by existing privacy-preserving approaches that lack a systematic enterprise data perspective. To bridge this gap, the paper introduces, for the first time, token-level and message-level differential privacy definitions tailored to generative agents. It formulates a probabilistic framework that models response generation as a randomized mechanism and derives privacy bounds dependent on both the softmax temperature and message length. Building on this foundation, the study quantitatively analyzes how generative parameters influence privacy leakage and proposes an optimized temperature configuration to achieve the best trade-off between privacy preservation and utility.

Technology Category

• Machine Learning: Privacy
• Natural Language Processing: Generation
• Philosophy and Ethics of AI: Privacy & Security

Application Category

• Security and Privacy: Privacy-enhancing technologies
• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large language models (LLMs) and AI agents are increasingly integrated into enterprise systems to access internal databases and generate context-aware responses. While such integration improves productivity and decision support, the model outputs may inadvertently reveal sensitive information. Although many prior efforts focus on protecting the privacy of user prompts, relatively few studies consider privacy risks from the enterprise data perspective. Hence, this paper develops a probabilistic framework for analyzing privacy leakage in AI agents based on differential privacy. We model response generation as a stochastic mechanism that maps prompts and datasets to distributions over token sequences. Within this framework, we introduce token-level and message-level differential privacy and derive privacy bounds that relate privacy leakage to generation parameters such as temperature and message length. We further formulate a privacy-utility design problem that characterizes optimal temperature selection.

Problem

• Differential Privacy
• Generative AI Agents
• Privacy Leakage
• Enterprise Data
• Large Language Models

Innovation

• differential privacy
• generative AI agents
• privacy-utility tradeoff
• token-level privacy
• stochastic generation mechanism