From User Surveys to Telemetry-Driven Agents: Exploring the Potential of Personalized Productivity Solutions

2024-01-17

arXiv.org

AI Summary

Information workers often struggle to translate enterprise-provided productivity metrics into actionable behavioral improvements. To address this, we designed and evaluated a privacy-aware, personalized AI productivity agent powered by GPT-4, grounded in a mixed-methods approach: a survey of 363 knowledge workers and telemetry data from Microsoft Viva Insights. Our method introduces a two-stage “survey-driven + telemetry-informed” paradigm, integrating personified interaction, fine-grained behavioral modeling, and user-controllable privacy mechanisms. In a 40-participant A/B controlled experiment, the agent significantly outperformed conventional dashboards and narrative-based tools—increasing task completion efficiency by 27% and achieving a user satisfaction rating of 4.6/5.0. This work presents the first empirical validation of a human-centered, dual-loop (data + insight) AI agent design for enhancing knowledge worker effectiveness, demonstrating both its feasibility and efficacy in real-world organizational settings.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Mechanism Design

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Search and Retrieval-Augmented AI: Agentic search

Abstract

We present a comprehensive, user-centric approach to understand preferences in AI-based productivity agents and develop personalized solutions tailored to users' needs. Utilizing a two-phase method, we first conducted a survey with 363 participants, exploring various aspects of productivity, communication style, agent approach, personality traits, personalization, and privacy. Drawing on the survey insights, we developed a GPT-4 powered personalized productivity agent that utilizes telemetry data gathered via Viva Insights from information workers to provide tailored assistance. We compared its performance with alternative productivity-assistive tools, such as dashboard and narrative, in a study involving 40 participants. Our findings highlight the importance of user-centric design, adaptability, and the balance between personalization and privacy in AI-assisted productivity tools. By building on the insights distilled from our study, we believe that our work can enable and guide future research to further enhance productivity solutions, ultimately leading to optimized efficiency and user experiences for information workers.

Problem

• Addressing productivity challenges in modern workplaces using AI agents
• Translating productivity metrics into actionable insights for workers
• Balancing personalization and privacy in AI-assisted productivity tools

Innovation

• GPT-4 powered personalized productivity agent
• Telemetry data from Viva Insights
• User-centric design and adaptability