🤖 AI Summary
This work proposes the first end-to-end system based on large language models (LLMs) to address the challenges of automated question answering for metascientific inquiries in scientometrics, such as academic entity recognition and multidimensional metric retrieval. By integrating retrieval-augmented generation (RAG) with an agent-based architecture, the system autonomously decomposes complex questions, plans analytical workflows, retrieves relevant data, and generates structured, high-level analyses. The approach innovatively leverages the reasoning and task-planning capabilities of LLMs within the domain of scientometrics and incorporates a proprietary research performance database. Expert evaluation and LLM-as-judge validation demonstrate that the system efficiently produces accurate, logically coherent analytical reports suitable for advanced scientometric inquiry.
📝 Abstract
This paper introduces AnalyticsGPT, an intuitive and efficient large language model (LLM)-powered workflow for scientometric question answering. This underrepresented downstream task addresses the subcategory of meta-scientific questions concerning the"science of science."When compared to traditional scientific question answering based on papers, the task poses unique challenges in the planning phase. Namely, the need for named-entity recognition of academic entities within questions and multi-faceted data retrieval involving scientometric indices, e.g. impact factors. Beyond their exceptional capacity for treating traditional natural language processing tasks, LLMs have shown great potential in more complex applications, such as task decomposition and planning and reasoning. In this paper, we explore the application of LLMs to scientometric question answering, and describe an end-to-end system implementing a sequential workflow with retrieval-augmented generation and agentic concepts. We also address the secondary task of effectively synthesizing the data into presentable and well-structured high-level analyses. As a database for retrieval-augmented generation, we leverage a proprietary research performance assessment platform. For evaluation, we consult experienced subject matter experts and leverage LLMs-as-judges. In doing so, we provide valuable insights on the efficacy of LLMs towards a niche downstream task. Our (skeleton) code and prompts are available at: https://github.com/lyvykhang/llm-agents-scientometric-qa/tree/acl.