From Queries to Criteria: Understanding How Astronomers Evaluate LLMs

2025-07-21

AI Summary

Current large language models (LLMs) lack domain-specific evaluation standards for astronomical research. Method: We propose a user-behavior-driven evaluation paradigm, grounded in 368 real-world astronomy queries from Slack and 11 in-depth user interviews. Using inductive coding combined with retrieval-augmented generation (RAG), we systematically identify astronomers’ usage patterns and core evaluation dimensions—including accuracy, traceability, and domain adaptability—for LLM-based literature interaction systems. Contribution/Results: We introduce AstroBench, the first extensible, astronomy-specific LLM benchmark, and derive science-oriented evaluation principles. AstroBench significantly enhances both the practical usability of LLMs in scientific workflows and the validity of their domain-specific assessment, establishing a methodological framework for developing evaluation benchmarks for specialized LLMs.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Evaluation and Analysis
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

There is growing interest in leveraging LLMs to aid in astronomy and other scientific research, but benchmarks for LLM evaluation in general have not kept pace with the increasingly diverse ways that real people evaluate and use these models. In this study, we seek to improve evaluation procedures by building an understanding of how users evaluate LLMs. We focus on a particular use case: an LLM-powered retrieval-augmented generation bot for engaging with astronomical literature, which we deployed via Slack. Our inductive coding of 368 queries to the bot over four weeks and our follow-up interviews with 11 astronomers reveal how humans evaluated this system, including the types of questions asked and the criteria for judging responses. We synthesize our findings into concrete recommendations for building better benchmarks, which we then employ in constructing a sample benchmark for evaluating LLMs for astronomy. Overall, our work offers ways to improve LLM evaluation and ultimately usability, particularly for use in scientific research.

Problem

• Understand how astronomers evaluate LLMs for scientific research
• Develop better benchmarks for LLM evaluation in astronomy
• Improve LLM usability through user feedback and criteria

Innovation

• Inductive coding of 368 LLM queries
• Deployed Slack bot for astronomy literature
• Constructed sample benchmark for LLM evaluation