From Canonical to Complex: Benchmarking LLM Capabilities in Undergraduate Thermodynamics

2025-08-29

AI Summary

This study investigates whether large language models (LLMs) can serve as unsupervised, principle-driven autonomous tutors for undergraduate thermodynamics instruction. Method: We construct UTQA—the first systematic, undergraduate-focused thermodynamics benchmark—comprising 50 questions spanning core topics including ideal gas processes, reversibility criteria, P–V diagram interpretation, and distinction between state and path functions; tasks are explicitly categorized into text-based reasoning and image-based reasoning. Contribution/Results: State-of-the-art LLMs achieve only up to 82% accuracy on text-based questions—below the educationally requisite 95% threshold—and perform near chance level (≈33%) on image-based reasoning, with poor robustness to prompt phrasing. These findings reveal fundamental limitations in LLMs’ ability to conduct consistent, first-principles–grounded thermodynamic reasoning and to integrate multimodal scientific representations, indicating they are currently unfit for reliable autonomous instruction in undergraduate thermodynamics.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Reasoning under Uncertainty: Uncertainty Representations
• Knowledge Representation and Reasoning: Automated Reasoning and Theorem Proving

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large language models (LLMs) are increasingly considered as tutoring aids in science education. Yet their readiness for unsupervised use in undergraduate instruction remains uncertain, as reliable teaching requires more than fluent recall: it demands consistent, principle-grounded reasoning. Thermodynamics, with its compact laws and subtle distinctions between state and path functions, reversibility, and entropy, provides an ideal testbed for evaluating such capabilities. Here we present UTQA, a 50-item undergraduate thermodynamics question answering benchmark, covering ideal-gas processes, reversibility, and diagram interpretation. No leading 2025-era model exceeded our 95% competence threshold: the best LLMs achieved 82% accuracy, with text-only items performing better than image reasoning tasks, which often fell to chance levels. Prompt phrasing and syntactic complexity showed modest to little correlation with performance. The gap concentrates in finite-rate/irreversible scenarios and in binding visual features to thermodynamic meaning, indicating that current LLMs are not yet suitable for unsupervised tutoring in this domain.

Problem

• Assessing LLM readiness for unsupervised thermodynamics tutoring
• Evaluating reasoning consistency in undergraduate thermodynamics education
• Identifying performance gaps in irreversible processes and visual interpretation

Innovation

• Benchmarking LLM capabilities in thermodynamics
• Evaluating unsupervised tutoring readiness
• Testing reasoning beyond fluent recall