Beliefs and Misconceptions around Integrated Conversational AI

2026-05-14

AI Summary

This study investigates how users develop an understanding of and trust in conversational AI systems deeply integrated into web browsers, such as Microsoft Edge’s Copilot. Through a qualitative user study combining task-based observations and semi-structured interviews, the research examines user interactions during information retrieval and planning tasks. Findings reveal that users’ trust in embedded conversational AI is largely shaped by their pre-existing mental models of large language models and traditional search engines. The inclusion of cited sources substantially enhances perceived output credibility, even when users do not actively verify them, and participants often rely on the same AI-provided references for fact-checking. These results demonstrate that citation mechanisms can effectively foster trust without requiring explicit user validation, offering empirical insights for the design of trustworthy AI systems.

Abstract

LLM-driven conversational AI is beginning to disappear into the background, shifting from something used directly towards something increasingly integrated into existing workflows. In the process, markers of origin and training are smoothed away as LLMs become commodified in the eyes of users. We explore how people approach using a web browser with conversational AI built in, focusing on how they develop their understanding and determine whether to trust its outputs. We conducted a study where 20 participants used the Copilot AI features in Microsoft Edge to conduct information retrieval and planning tasks. Participants relied on a combination of existing perceptions of LLMs and internet search, tracing the effect of beliefs about how Copilot generated answers on prompting strategies. The inclusion of citations increased the trustworthiness of answers without participants feeling the need to be check them, with participants often reaching for the same information sources as the CAI when fact-checking.

Problem

• Conversational AI
• Trust
• User Beliefs
• LLM Integration
• Information Retrieval

Innovation

• Conversational AI
• Trust in AI
• Large Language Models
• Human-AI Interaction
• Information Retrieval