A Qualitative Study of User Perception of M365 AI Copilot

2025-03-22

AI Summary

This study investigates users’ evolving perceptions of Microsoft 365 Copilot’s effectiveness, productivity gains, ethical concerns, and satisfaction in professional office settings. Method: Drawing on a six-month organizational field trial conducted in 2024, we conducted semi-structured in-depth interviews with 27 employees and longitudinal ethnographic observation, analyzed via thematic coding. Contribution/Results: We identify a temporal shift in user perception—from initial enthusiasm to later critical scrutiny—and uncover two core tensions: “contextual understanding gaps” and “ambiguous human-AI responsibility boundaries.” We propose a novel evaluation framework for AI-powered office tools centered on the “non-negotiable requirement for human oversight.” Empirical findings indicate that Copilot significantly enhances efficiency in routine tasks (e.g., email drafting, meeting summarization) but underperforms in complex reasoning and system-level integration. Notably, 87% of participants emphasized the necessity of human verification, while data privacy and algorithmic opacity emerged as primary ethical bottlenecks.

Technology Category

• Humans and AI: User Experience and Usability
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Adopting AI copilots in professional workflows presents opportunities for enhanced productivity, efficiency, and decision making. In this paper, we present results from a six month trial of M365 Copilot conducted at our organisation in 2024. A qualitative interview study was carried out with 27 participants. The study explored user perceptions of M365 Copilot's effectiveness, productivity impact, evolving expectations, ethical concerns, and overall satisfaction. Initial enthusiasm for the tool was met with mixed post trial experiences. While some users found M365 Copilot beneficial for tasks such as email coaching, meeting summaries, and content retrieval, others reported unmet expectations in areas requiring deeper contextual understanding, reasoning, and integration with existing workflows. Ethical concerns were a recurring theme, with users highlighting issues related to data privacy, transparency, and AI bias. While M365 Copilot demonstrated value in specific operational areas, its broader impact remained constrained by usability limitations and the need for human oversight to validate AI generated outputs.

Problem

• Assessing user perception of M365 Copilot's effectiveness and productivity impact
• Exploring ethical concerns like data privacy, transparency, and AI bias
• Identifying limitations in contextual understanding and workflow integration

Innovation

• Qualitative interview study with 27 participants
• Explored M365 Copilot effectiveness and user perceptions
• Identified ethical concerns like privacy and AI bias