A systematic evaluation of vision-language models for observational astronomical reasoning tasks

2026-04-27

AI Summary

This study addresses the limited reliability of vision-language models (VLMs) in multimodal astronomical tasks by introducing AstroVLBench, a benchmark comprising over 4,100 expert-validated samples spanning five key domains: optical imaging, radio interferometry, multi-band photometry, time-domain light curves, and spectroscopy. The work presents the first systematic evaluation of mainstream VLMs on these tasks and investigates their reasoning mechanisms through ablation studies, comparisons between physics-based and phenomenological prompts, and direct numerical table inputs. Results reveal that general-purpose VLMs significantly underperform domain-specific methods, with Gemini 3 Pro demonstrating the most consistent performance. Incorporating physics-guided prompts or feeding raw numerical data improves accuracy by up to 13 percentage points and mitigates class bias, underscoring the critical role of grounding models in physical knowledge for robust astronomical reasoning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Vision-language models (VLMs) are increasingly proposed as general-purpose tools for scientific data interpretation, yet their reliability on real astronomical observations across diverse modalities remains untested. We present AstroVLBench, a comprehensive benchmark comprising over 4,100 expert-verified instances across five tasks spanning optical imaging, radio interferometry, multi-wavelength photometry, time-domain light curves, and optical spectroscopy. Evaluating six frontier models, we find that performance is strongly modality-dependent: while one model (Gemini 3 Pro) emerges as the most consistently capable across tasks, task-specific strengths vary, and all models substantially underperform domain-specialized methods. Mechanistic ablations reveal that performance depends not only on directing attention to salient visual features but also on grounding those features in physical knowledge. Phenomenological prompts describing what to look for improve accuracy by sharpening model focus, but physical prompts explaining why those features matter perform better overall and yield more balanced classifications with reduced class-specific bias. Consistent with this picture, presenting the underlying one-dimensional measurements directly as numerical tables instead of rendered plots yields up to 13 percentage points improvement. Reasoning quality analysis further demonstrates that, without explicit physical grounding, models may reach correct predictions from phenomenologically plausible cues while providing physically imprecise justifications, establishing that accuracy alone is insufficient for trustworthy scientific deployment. These findings provide the first systematic, multi-modal baselines for VLMs in observational astronomy and identify the specific representation, grounding, and reasoning bottlenecks where current models fail.

Problem

• vision-language models
• observational astronomy
• scientific reasoning
• multi-modal evaluation
• model reliability

Innovation

• vision-language models
• observational astronomy
• physical grounding
• multi-modal benchmark
• scientific reasoning