"Can You Tell Me?": Designing Copilots to Support Human Judgement in Online Information Seeking

2026-01-16

AI Summary

This study addresses the risk that generative AI in information retrieval may foster user overreliance, thereby undermining critical thinking and independent verification skills. To counter this, the authors propose a large language model–based conversational collaborator that eschews direct answers in favor of Socratic questioning, employing cognitive scaffolding to prompt users to reflect on the credibility of information and cultivate digital literacy. Evaluated through a randomized controlled trial and mixed-methods analysis, the system elicited significantly enhanced metacognitive reflection among users. However, it did not yield measurable improvements in answer accuracy or search engagement, highlighting an inherent tension between efficiency-oriented search behaviors and the cultivation of critical information literacy. This approach offers a novel paradigm that supports—rather than supplants—user judgment.

Technology Category

• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Search and Optimization: Metareasoning and Metaheuristics
• Humans and AI: Intelligent User Interfaces

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Generative AI (GenAI) tools are transforming information seeking, but their fluent, authoritative responses risk overreliance and discourage independent verification and reasoning. Rather than replacing the cognitive work of users, GenAI systems should be designed to support and scaffold it. Therefore, this paper introduces an LLM-based conversational copilot designed to scaffold information evaluation rather than provide answers and foster digital literacy skills. In a pre-registered, randomised controlled trial (N=261) examining three interface conditions including a chat-based copilot, our mixed-methods analysis reveals that users engaged deeply with the copilot, demonstrating metacognitive reflection. However, the copilot did not significantly improve answer correctness or search engagement, largely due to a"time-on-chat vs. exploration"trade-off and users'bias toward positive information. Qualitative findings reveal tension between the copilot's Socratic approach and users'desire for efficiency. These results highlight both the promise and pitfalls of pedagogical copilots, and we outline design pathways to reconcile literacy goals with efficiency demands.

Problem

• Generative AI
• information seeking
• overreliance
• digital literacy
• cognitive scaffolding

Innovation

• conversational copilot
• generative AI
• scaffolding
• digital literacy
• metacognitive reflection