Reasoning over mathematical objects: on-policy reward modeling and test time aggregation
🤖 AI Summary
This work addresses the limitation of current large language models (LLMs) in STEM domains, where they predominantly generate simplified outputs such as numerical values or multiple-choice answers and struggle to produce structured mathematical objects. To overcome this, the authors introduce Principia, a novel dataset and benchmark that systematically supports the generation of formal mathematical structures. They propose an online policy reward modeling framework that jointly trains LLMs with strong model-based judges and verifiers, augmented by test-time inference aggregation to leverage extended computational resources. The approach substantially improves performance across diverse backbone models on structured mathematical reasoning tasks, while also achieving superior results on conventional numerical and multiple-choice benchmarks, thereby demonstrating both its effectiveness and cross-format generalization capability.
Application Category
📝 Abstract
The ability to precisely derive mathematical objects is a core requirement for downstream STEM applications, including mathematics, physics, and chemistry, where reasoning must culminate in formally structured expressions. Yet, current LM evaluations of mathematical and scientific reasoning rely heavily on simplified answer formats such as numerical values or multiple choice options due to the convenience of automated assessment. In this paper we provide three contributions for improving reasoning over mathematical objects: (i) we build and release training data and benchmarks for deriving mathematical objects, the Principia suite; (ii) we provide training recipes with strong LLM-judges and verifiers, where we show that on-policy judge training boosts performance; (iii) we show how on-policy training can also be used to scale test-time compute via aggregation. We find that strong LMs such as Qwen3-235B and o3 struggle on Principia, while our training recipes can bring significant improvements over different LLM backbones, while simultaneously improving results on existing numerical and MCQA tasks, demonstrating cross-format generalization of reasoning abilities.
Problem
Research questions and friction points this paper is trying to address.
mathematical reasoning
structured expressions
LLM evaluation
STEM applications
answer formats
Innovation
Methods, ideas, or system contributions that make the work stand out.
on-policy reward modeling
mathematical reasoning
test-time aggregation
LLM judges
Principia benchmark
🔎 Similar Papers
2024-05-31arXiv.orgCitations: 2
