Understanding Usage and Engagement in AI-Powered Scientific Research Tools: The Asta Interaction Dataset
🤖 AI Summary
This study addresses the current lack of systematic understanding of how researchers use AI-powered scientific tools in real-world settings. The authors construct and analyze the Asta dataset, comprising over 200,000 user interaction logs, to reveal—for the first time—that researchers treat large language model (LLM)-enhanced retrieval-augmented generation (RAG) platforms as collaborative partners. Through query intent classification, interaction sequence modeling, and log analysis, the study finds that users tend to submit longer, more complex queries and reuse generated content as persistent research artifacts. With increased experience, their queries become more focused and exhibit greater reliance on citations, although keyword-based queries remain prevalent. This work introduces a novel query intent taxonomy and releases the first large-scale dataset of authentic researcher–AI interactions.
Application Category
📝 Abstract
AI-powered scientific research tools are rapidly being integrated into research workflows, yet the field lacks a clear lens into how researchers use these systems in real-world settings. We present and analyze the Asta Interaction Dataset, a large-scale resource comprising over 200,000 user queries and interaction logs from two deployed tools (a literature discovery interface and a scientific question-answering interface) within an LLM-powered retrieval-augmented generation platform. Using this dataset, we characterize query patterns, engagement behaviors, and how usage evolves with experience. We find that users submit longer and more complex queries than in traditional search, and treat the system as a collaborative research partner, delegating tasks such as drafting content and identifying research gaps. Users treat generated responses as persistent artifacts, revisiting and navigating among outputs and cited evidence in non-linear ways. With experience, users issue more targeted queries and engage more deeply with supporting citations, although keyword-style queries persist even among experienced users. We release the anonymized dataset and analysis with a new query intent taxonomy to inform future designs of real-world AI research assistants and to support realistic evaluation.
Problem
Research questions and friction points this paper is trying to address.
AI-powered scientific research tools
user interaction
query patterns
engagement behavior
real-world usage
Innovation
Methods, ideas, or system contributions that make the work stand out.
interaction dataset
retrieval-augmented generation
query intent taxonomy
AI research assistant
user behavior analysis
🔎 Similar Papers
No similar papers found.
