Designing and Evaluating AI Margin Notes in Document Reader Software

2025-09-11

AI Summary

This paper addresses the contextual fragmentation inherent in AI-augmented reading, where standalone chat interfaces decouple AI assistance from document content. We propose “AI sidebar annotations”—a novel paradigm embedding AI functionality directly within the document’s margin space. First, we systematically construct the design space for AI sidebar annotations. Then, through multiple controlled experiments, we investigate how integration modality, text selection mechanism, and human-AI collaboration intensity affect user experience. Results indicate strong user preference for integrated (vs. detached) presentation and manual (vs. automatic) text selection. Low AI involvement enhances perceived psychological ownership, whereas moderate involvement achieves optimal trade-offs between task efficiency and user acceptance. Notably, AI intervention level shows no significant effect on reading comprehension—challenging the prevailing “more AI is better” assumption. Our findings provide both theoretical grounding and actionable design guidelines for AI-native document tools.

Technology Category

• Humans and AI: Intelligent User Interfaces
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction
• Intelligent Robots: Embodied AI

Application Category

• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Search and Retrieval-Augmented AI: Agentic search
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

AI capabilities for document reader software are usually presented in separate chat interfaces. We explore integrating AI into document comments, a concept we formalize as AI margin notes. Three design parameters characterize this approach: margin notes are integrated with the text while chat interfaces are not; selecting text for a margin note can be automated through AI or manual; and the generation of a margin note can involve AI to various degrees. Two experiments investigate integration and selection automation, with results showing participants prefer integrated AI margin notes and manual selection. A third experiment explores human and AI involvement through six alternative techniques. Techniques with less AI involvement resulted in more psychological ownership, but faster and less effortful designs are generally preferred. Surprisingly, the degree of AI involvement had no measurable effect on reading comprehension. Our work shows that AI margin notes are desirable and contributes implications for their design.

Problem

• Integrating AI into document comments as margin notes
• Comparing automated versus manual text selection methods
• Evaluating effects of AI involvement on user experience

Innovation

• Integrated AI margin notes in documents
• Automated or manual text selection methods
• Varied AI involvement levels for generation