Is Passive Expertise-Based Personalization Enough? A Case Study in AI-Assisted Test-Taking

2025-11-28

AI Summary

This study investigates whether passive personalization—i.e., static adaptation solely based on users’ domain expertise (expert vs. novice)—effectively enhances user experience and task performance in AI-augmented high-stakes examination settings. We developed an enterprise-grade AI assistant with passive personalization capabilities and conducted a controlled user study featuring timed exam tasks, comparing system variants. Results show that passive personalization significantly reduces cognitive load and improves perceived trustworthiness and usability of the AI assistant; however, it yields only marginal gains in complex task performance, revealing a task-specificity bottleneck. Our key contribution is the first empirical demonstration of the dual effect of passive personalization in high-stakes human-AI collaboration: broad benefits in subjective metrics versus inherent limitations in objective task outcomes. We further propose augmenting passive personalization with active personalization mechanisms to strengthen user agency and adaptive control.

Technology Category

• Humans and AI: User Experience and Usability
• Cognitive Modeling & Cognitive Systems: Adaptive Behavior
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Semantics and Knowledge: Applications of semantic technologies for improving search, browsing, recommendation, personalization

Abstract

Novice and expert users have different systematic preferences in task-oriented dialogues. However, whether catering to these preferences actually improves user experience and task performance remains understudied. To investigate the effects of expertise-based personalization, we first built a version of an enterprise AI assistant with passive personalization. We then conducted a user study where participants completed timed exams, aided by the two versions of the AI assistant. Preliminary results indicate that passive personalization helps reduce task load and improve assistant perception, but reveal task-specific limitations that can be addressed through providing more user agency. These findings underscore the importance of combining active and passive personalization to optimize user experience and effectiveness in enterprise task-oriented environments.

Problem

• Investigating whether expertise-based personalization improves user experience in AI-assisted tasks
• Examining effects of passive personalization on task load and assistant perception
• Exploring limitations of passive personalization and need for user agency

Innovation

• Passive personalization reduces user task load
• Active personalization addresses task-specific limitations
• Combining personalization methods optimizes user experience