Autonomy Matters: A Study on Personalization-Privacy Dilemma in LLM Agents

2025-10-05

AI Summary

This study investigates the interactive effects of LLM agent autonomy and personalization on users’ privacy concerns, trust, and adoption intention—aiming to alleviate the inherent tension between personalization and privacy protection. A 3×3 between-subjects experiment (N=450) integrated psychological mechanism analysis. Results show that personalization that disregards users’ privacy preferences significantly heightens privacy concerns and erodes trust; however, moderate agent autonomy effectively mitigates these adverse effects, enhancing user acceptance and trust. Critically, autonomy exhibits a nonlinear moderating effect: both low and high autonomy impair outcomes, whereas an intermediate level optimizes them. The study’s key contribution is conceptualizing “controllable autonomy”—a design paradigm granting agents bounded decision-making authority while preserving salient user control points—to dynamically balance privacy preservation and personalization utility.

Technology Category

• Machine Learning: Privacy
• Multiagent Systems: Mechanism Design
• Humans and AI: Learning Human Values and Preferences

Application Category

• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large Language Model (LLM) agents require personal information for personalization in order to better act on users' behalf in daily tasks, but this raises privacy concerns and a personalization-privacy dilemma. Agent's autonomy introduces both risks and opportunities, yet its effects remain unclear. To better understand this, we conducted a 3$ imes$3 between-subjects experiment ($N=450$) to study how agent's autonomy level and personalization influence users' privacy concerns, trust and willingness to use, as well as the underlying psychological processes. We find that personalization without considering users' privacy preferences increases privacy concerns and decreases trust and willingness to use. Autonomy moderates these effects: Intermediate autonomy flattens the impact of personalization compared to No- and Full autonomy conditions. Our results suggest that rather than aiming for perfect model alignment in output generation, balancing autonomy of agent's action and user control offers a promising path to mitigate the personalization-privacy dilemma.

Problem

• Investigates the personalization-privacy dilemma in LLM agents
• Examines how autonomy level affects user trust and privacy concerns
• Explores balancing agent autonomy with user control to resolve dilemma

Innovation

• Balancing agent autonomy with user control
• Intermediate autonomy moderates personalization effects
• Experimental study on personalization-privacy dilemma mitigation