Blending Queries and Conversations: Understanding Tactics, Trust, Verification, and System Choice in Web Search and Chat Interactions

2025-04-07

AI Summary

This study investigates user behavior, trust dynamics, and verification strategies during health information retrieval in hybrid interfaces integrating web search and generative AI chat. Through think-aloud protocols with 22 participants, combined with qualitative coding and quantitative behavioral analysis, we systematically identify 78 distinct user tactics and uncover a “trust mismatch” phenomenon: users overtrust AI outputs due to interface usability and superficial answer completeness, leading to paradoxically heightened confidence in incorrect responses. Key findings include: (1) pre-task confidence and trust jointly influence interface selection; (2) active verification is markedly insufficient during chat-based interaction; and (3) generative AI is neither a universal replacement for nor substantially inferior to traditional search. The work provides foundational human factors insights and actionable guidelines for designing trustworthy hybrid search systems and deploying reliable AI in health information contexts.

Technology Category

• Humans and AI: Intelligent User Interfaces
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

This paper presents a user study (N=22) where participants used an interface combining Web Search and a Generative AI-Chat feature to solve health-related information tasks. We study how people behaved with the interface, why they behaved in certain ways, and what the outcomes of these behaviours were. A think-aloud protocol captured their thought processes during searches. Our findings suggest that GenAI is neither a search panacea nor a major regression compared to standard Web Search interfaces. Qualitative and quantitative analyses identified 78 tactics across five categories and provided insight into how and why different interface features were used. We find evidence that pre-task confidence and trust both influenced which interface feature was used. In both systems, but particularly when using the chat feature, trust was often misplaced in favour of ease-of-use and seemingly perfect answers, leading to increased confidence post-search despite having incorrect results. We discuss what our findings mean in the context of our defined research questions and outline several open questions for future research.

Problem

• Understanding user behavior in Web Search and AI-Chat interactions
• Examining trust and confidence influences on interface choice
• Analyzing tactics and outcomes in health information searches

Innovation

• Combined Web Search and Generative AI-Chat interface
• Think-aloud protocol for user behavior analysis
• Qualitative and quantitative analysis of 78 tactics