Reason, Reward, Refine: Step-Level Errors Corrections with Structured Feedback for Physics Reasoning in Small Language Models

2026-07-06

AI Summary

This work addresses the vulnerability of small language models to cascading failures in physical reasoning, which often stem from single-step errors due to insufficient domain knowledge, multi-step hallucinations, and sensitivity to distribution shifts. To mitigate this, the authors propose a stepwise reward framework that leverages an external verifier during training to identify the first erroneous step, generate structured feedback, and optimize the model’s reasoning trajectory via KL-regularized policy gradient—without requiring ground-truth answers or human preference labels. Relying solely on the external verifier for step-level correction, the method achieves 17–20% absolute accuracy gains over chain-of-thought (CoT) prompting and 10–16% improvements over the strongest baseline across five physics benchmarks. Furthermore, it reduces computational and misinterpretation error rates from 56.9% and 22.3% to 23.5% and 12.0%, respectively.

Abstract

Physics reasoning fails structurally in small language models: an error at any step propagates forward, corrupting every inference that follows. Limited domain knowledge, hallucination under multi-step derivation, and distributional sensitivity compound this failure. We propose a step-level reward framework that identifies the first reasoning error, generates targeted structured feedback, and trains the model to revise its solution via policy gradient with KL regularization, without exposing it to ground truth solutions as generation targets. Unlike annotation-dependent step-level methods, no preference data construction is required and the external verifier operates exclusively at training time. Across five physics benchmarks, our framework delivers accuracy gains of 17-20% over CoT prompting and 10-16% over the strongest baseline, reduces calculation errors from 56.9% to 23.5%, and reduces miscomprehension errors from 22.3% to 12.0% in the best observed cases. Conceptual errors reduce from 89.7% to 68.7%, yet persist as the hardest failure mode across all conditions.

Problem

• physics reasoning
• step-level errors
• small language models
• error propagation
• structured feedback

Innovation

• step-level reward
• structured feedback
• physics reasoning
• small language models
• error correction