ExpStar: Towards Automatic Commentary Generation for Multi-discipline Scientific Experiments

2025-07-13

AI Summary

This study addresses the time-consuming, expertise-intensive nature of manually authoring multidisciplinary scientific experiment commentaries. We propose the first automated framework for experiment commentary generation. Methodologically, we construct ExpInstruct—a multidisciplinary dataset comprising over 7,000 high-quality instruction-following samples—and design ExpStar, a retrieval-augmented generation (RAG) model that jointly models procedural descriptions, scientific principle explanations (e.g., chemical equations, physical laws), and safety protocols. Our contributions include: (1) establishing the first cross-disciplinary benchmark dataset for experiment commentary; and (2) introducing a knowledge-aware, step-level retrieval mechanism to enhance commentary professionalism and granularity. Extensive evaluation across 14 state-of-the-art large language models demonstrates that ExpStar significantly outperforms existing approaches, validating both the dataset’s quality and the architectural design. This work provides a scalable, AI-driven technical pathway for enhancing experimental science education.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Application Domains: Humanities & Computational Social Science
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Experiment commentary is crucial in describing the experimental procedures, delving into underlying scientific principles, and incorporating content-related safety guidelines. In practice, human teachers rely heavily on subject-specific expertise and invest significant time preparing such commentary. To address this challenge, we introduce the task of automatic commentary generation across multi-discipline scientific experiments. While recent progress in large multimodal models (LMMs) has demonstrated promising capabilities in video understanding and reasoning, their ability to generate fine-grained and insightful experiment commentary remains largely underexplored. In this paper, we make the following contributions: (i) We construct extit{ExpInstruct}, the first dataset tailored for experiment commentary generation, featuring over 7 extit{K} step-level commentaries across 21 scientific subjects from 3 core disciplines (ie, science, healthcare and engineering). Each sample includes procedural descriptions along with potential scientific principles (eg, chemical equations and physical laws) and safety guidelines. (ii) We propose ExpStar, an automatic experiment commentary generation model that leverages a retrieval-augmented mechanism to adaptively access, evaluate, and utilize external knowledge. (iii) Extensive experiments show that our ExpStar substantially outperforms 14 leading LMMs, which highlights the superiority of our dataset and model. We believe that ExpStar holds great potential for advancing AI-assisted scientific experiment instruction.

Problem

• Automatic commentary generation for multi-discipline experiments
• Lack of fine-grained experiment analysis in current LMMs
• Need for dataset and model for scientific experiment instruction

Innovation

• Constructs ExpInstruct dataset for experiment commentary
• Proposes retrieval-augmented ExpStar model
• Outperforms 14 leading LMMs in experiments