Data Discovery using LLMs -- A Study of Data User Behaviour

2025-07-06

AI Summary

This study investigates the applicability of large language models (LLMs) for scientific data search and examines researchers’ interaction behaviors and experiences. A two-phase empirical study (N=32) employed persona-based prompting as an experimental intervention, integrating behavioral log analysis, semi-structured interviews, and task performance evaluation. Results indicate that users predominantly formulate queries in natural language but perceive LLMs as instrumental tools rather than conversational partners. Persona prompting significantly improved perceived usefulness and interaction satisfaction only among participants with prior LLM experience. This work provides the first empirical evidence of user experience level as a critical moderator of prompt engineering effectiveness. It thereby advances human-centered design for scholarly data retrieval systems by identifying cognition-informed optimization pathways grounded in empirically validated user heterogeneity.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Humans and AI: User Experience and Usability
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Data search for scientific research is more complex than a simple web search. The emergence of large language models (LLMs) and their applicability for scientific tasks offers new opportunities for researchers who are looking for data, e.g., to freely express their data needs instead of fitting them into restrictions of data catalogues and portals. However, this also creates uncertainty about whether LLMs are suitable for this task. To answer this question, we conducted a user study with 32 researchers. We qualitatively and quantitively analysed participants' information interaction behaviour while searching for data using LLMs in two data search tasks, one in which we prompted the LLM to behave as a persona. We found that participants interact with LLMs in natural language, but LLMs remain a tool for them rather than an equal conversational partner. This changes slightly when the LLM is prompted to behave as a persona, but the prompting only affects participants' user experience when they are already experienced in LLM use.

Problem

• Evaluating LLMs' suitability for complex scientific data search
• Analyzing user behavior when using LLMs for data discovery
• Assessing impact of persona-prompting on LLM search experience

Innovation

• Using LLMs for natural language data search
• Analyzing user behavior with qualitative and quantitative methods
• Exploring persona prompting effects on user experience