AI Personalization Paradox: Personalized AI Increases Superficial Engagement in Reading while Undermines Autonomy and Ownership in Writing

2026-01-25

AI Summary

This study addresses a critical tension in personalized AI writing assistance: while such systems enhance surface-level engagement, they may inadvertently erode authors’ creative autonomy and sense of textual ownership. The research identifies a novel “personalization paradox”—an AI system tailored to users’ prior reading behaviors (e.g., highlighted passages) increases the frequency of highlight usage during writing but encourages instrumental, rather than reflective, annotation practices, fostering overreliance on AI. Through a between-subjects experiment integrating behavioral logs and qualitative interviews, the findings demonstrate that personalization significantly diminishes users’ perceived autonomy, ownership, and self-identification with their writing. These results reveal a profound risk of cognitive alienation when writing support is driven primarily by past reading behaviors, highlighting the need for more cognitively aligned personalization strategies.

Technology Category

• Humans and AI: Intelligent User Interfaces
• Cognitive Modeling & Cognitive Systems: Computational Creativity
• Natural Language Processing: Sentiment Analysis, Stylistic Analysis, and Argument Mining

Application Category

• User Modeling, Personalization and Recommendation: Studies of user behavior, including longitudinal effects of personalized systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search

Abstract

AI-assisted writing raises concerns about autonomy and ownership when benefiting writers. Personalization has been proposed as an effective solution while also risking writers'reliance on AI and behavior shifting. For better personalization design, existing studies rely on interaction and information solely within the writing phase; however, few studies have examined how reading behaviors can inform personalized writing. This study investigates the effects of integrating reading highlights for personalization on AI-assisted writing. A between-subjects study with 46 participants revealed that the personalization condition encouraged participants to produce more highlights. However, highlighting unexpectedly shifted from a sense-making strategy to an instrumental act of"feeding the AI,"leading to significant reliance on AI and declines in writers'sense of autonomy, ownership, and self-credit. These findings indicate personalization risks in AI-assisted writing, emphasize the importance of personalization strategies, and provide design implications.

Problem

• AI personalization
• autonomy
• ownership
• AI-assisted writing
• reading behavior

Innovation

• AI personalization
• reading-to-writing integration
• autonomy in writing
• highlighting behavior
• AI-assisted writing