World Models for Math Story Problems

2023-06-07

Annual Meeting of the Association for Computational Linguistics

AI Summary

Large language models (LLMs) suffer from semantic misinterpretation and insufficient interpretability when solving mathematical word problems. Method: We propose MathWorld—the first world-model formalism tailored for mathematical story problems—explicitly representing real-world scenarios, actions, and mathematical relations via a graph-structured knowledge representation. It integrates logic-form annotations (1,019 problems, 3,204 logic forms) with graph neural network–based semantic modeling to achieve structured conceptual representation. Innovatively, it unifies interpretable reasoning analysis and controllable problem generation through synthetic question-answering prompting and world-model–guided constrained question synthesis. Contribution/Results: Experiments demonstrate that MathWorld effectively supports LLM reasoning evaluation and generates high-quality, semantically coherent novel problems. It significantly enhances models’ depth of mathematical semantic understanding and decision transparency, establishing a foundation for interpretable, controllable, and semantically grounded mathematical reasoning.

Abstract

Solving math story problems is a complex task for students and NLP models alike, requiring them to understand the world as described in the story and reason over it to compute an answer. Recent years have seen impressive performance on automatically solving these problems with large pre-trained language models and innovative techniques to prompt them. However, it remains unclear if these models possess accurate representations of mathematical concepts. This leads to lack of interpretability and trustworthiness which impedes their usefulness in various applications. In this paper, we consolidate previous work on categorizing and representing math story problems and develop MathWorld, which is a graph-based semantic formalism specific for the domain of math story problems. With MathWorld, we can assign world models to math story problems which represent the situations and actions introduced in the text and their mathematical relationships. We combine math story problems from several existing datasets and annotate a corpus of 1,019 problems and 3,204 logical forms with MathWorld. Using this data, we demonstrate the following use cases of MathWorld: (1) prompting language models with synthetically generated question-answer pairs to probe their reasoning and world modeling abilities, and (2) generating new problems by using the world models as a design space.

Problem

• Develop MathWorld for math story problems
• Probe models' reasoning and world modeling abilities
• Generate new problems using world models

Innovation

• Graph-based semantic formalism
• Annotated corpus with MathWorld
• Synthetic Q&A for probing models