The AlphaPhysics Term Rewriting System for Marking Algebraic Expressions in Physics Exams

2025-07-24

AI Summary

Automated assessment of algebraic expressions in physics examinations poses significant challenges due to semantic variability, contextual dependencies, and the need for domain-aware equivalence reasoning. Method: This paper proposes a semantic equivalence verification framework integrating domain-specific term rewriting, Satisfiability Modulo Theories (SMT) solving, and large language models (LLMs). It introduces a terminating and confluent term rewriting system tailored to physics contexts; leverages computer algebra systems (CAS) for handling transcendental functions (e.g., trigonometric expressions); employs SMT solvers for precise symbolic reasoning; and utilizes LLMs to align natural-language answers with formal expressions and correct semantic mismatches. Contribution/Results: Evaluated on over 1,500 real student responses from the 2023 Australian Physics Olympiad, the framework substantially outperforms pure symbolic or LLM-only baselines. It achieves high accuracy while enabling interpretable, formally verifiable automated grading—bridging the gap between robust symbolic reasoning and flexible linguistic understanding in physics education assessment.

Technology Category

• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Knowledge Representation and Reasoning: Automated Reasoning and Theorem Proving
• Computer Vision: Low Level & Physics-based Vision

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

We present our method for automatically marking Physics exams. The marking problem consists in assessing typed student answers for correctness with respect to a ground truth solution. This is a challenging problem that we seek to tackle using a combination of a computer algebra system, an SMT solver and a term rewriting system. A Large Language Model is used to interpret and remove errors from student responses and rewrite these in a machine readable format. Once formalized and language-aligned, the next step then consists in applying automated reasoning techniques for assessing student solution correctness. We consider two methods of automated theorem proving: off-the-shelf SMT solving and term rewriting systems tailored for physics problems involving trigonometric expressions. The development of the term rewrite system and establishing termination and confluence properties was not trivial, and we describe it in some detail in the paper. We evaluate our system on a rich pool of over 1500 real-world student exam responses from the 2023 Australian Physics Olympiad.

Problem

• Automatically marking Physics exams by assessing student answers
• Using computer algebra and SMT solvers for correctness evaluation
• Formalizing student responses with LLMs for automated reasoning

Innovation

• Combines computer algebra system, SMT solver, term rewriting
• Uses Large Language Model to interpret student responses
• Tailored term rewriting for trigonometric physics problems