PrivacyAlign: Contextual Privacy Alignment for LLM Agents

2026-06-19

AI Summary

Large language model agents often pose privacy risks when acting on behalf of users due to insufficient understanding of context-sensitive privacy boundaries. This work addresses this challenge by constructing a high-quality, human-annotated dataset grounded in real-world privacy judgments from a diverse population and introduces a label-conditioned reward modeling approach. Specifically, the method trains agents to align with human privacy norms by conditioning a large-model-based critic within a reinforcement learning framework on fine-grained annotation labels. Experimental results demonstrate that small, open-source agents trained with this approach significantly outperform baseline models on both the PrivacyAlign benchmark and existing privacy evaluation suites, exhibiting a markedly improved ability to adhere to socially agreed-upon privacy boundaries.

Abstract

AI agents acting on behalf of users are constantly making decisions, and for users to trust their agents, those decisions must align with what they actually want. Privacy is an important alignment problem for agents: every message, post, or tool call an agent makes is a contextual judgment about what is appropriate to share, with whom, and under which conditions. Because such judgments depend on social expectations and norms, human judgment does not merely label privacy violations but also helps define them. While existing work relies on unreliable proxies for both training and evaluation, we place human judgment at the center of agentic privacy alignment. We introduce PrivacyAlign, a dataset of 1,350 samples with 3,516 detailed annotations from 599 unique annotators across diverse scenarios where current LLMs actually leak, and use it to ground both alignment training and automated evaluation in human privacy norms. Building on these annotations, we first show that conditioning LLM judges on human annotations and explanations for reference responses to the same prompt makes their judgments more reliable. We then introduce annotation-conditioned reward modeling, which uses these annotations to score new responses during RL, and show that small open-weight agents trained with this reward better align with human privacy norms, with strong gains on PrivacyAlign and existing privacy benchmarks for agents.

Problem

• privacy alignment
• LLM agents
• human judgment
• privacy norms
• contextual privacy

Innovation

• Privacy Alignment
• Human-in-the-loop Annotation
• Annotation-conditioned Reward Modeling
• LLM Agents
• Contextual Privacy