SciText2Eq: Assessing LLMs for Explainable Equation Generation for Scientific Creativity

2026-06-14

AI Summary

This study addresses key challenges in large language models’ generation of mathematical equations within scientific texts, including difficulties in semantic alignment, inadequate modeling of dependencies among multiple equations, and misalignment between existing evaluation metrics and human judgment. To tackle these issues, the authors construct a high-quality dataset comprising contextual paragraphs, ground-truth equations, and variable descriptions, propose an interpretable equation generation pipeline, and introduce a comprehensive evaluation protocol that integrates automatic metrics, large model–based scoring, and human assessment. Experimental results reveal that while current models perform reasonably well at lexical and syntactic levels, they exhibit significant deficiencies in semantic accuracy, and existing automatic evaluations show limited agreement with human judgments. This work provides the first systematic benchmark for assessing equation generation in terms of accuracy, interpretability, and human-model consistency, offering both methodological and empirical foundations for future research.

Abstract

This work investigates the ability of large language models (LLMs) to generate mathematical equations from scientific texts. Prior work faces challenges in unstructured grounding, multi-equation dependency, and humanaligned evaluation. To this end, we construct a dataset of AI research papers, pairing contextual passages with ground-truth equations and variable descriptions. We develop an explainable equation generation workflow and evaluate it across diverse open- and closed-source LLM backbones. We introduce an evaluation protocol combining automatic metrics, LLM-based rubrics, and human judgments to assess accuracy, explainability, and human-LLM alignment. Results indicate that LLMs perform moderately on lexical- and syntactic-based similarity, while struggling with semantic accuracy. Comparisons between LLM-based evaluations and human judgments reveal limited alignment, highlighting challenges in using LLMs to assess equation quality. These findings offer insights for improving equation generation models and developing more reliable evaluation methods for scientific text. We provide code and data for reproducibility.

Problem

• equation generation
• large language models
• scientific text
• explainability
• evaluation alignment

Innovation

• explainable equation generation
• scientific text-to-equation
• LLM evaluation
• human-LLM alignment
• semantic accuracy