Reinforcing Step-level Reasoning for Effective Self-Correction in LLMs

2026-08-11

AI Summary

Large language models struggle to autonomously identify and correct reasoning errors. To address this limitation, this work proposes the Self-Fix Step-DPO (SFS-DPO) framework, which uniquely integrates step-level preference optimization with explicit self-correction training. The approach operates in two stages: first, Step-DPO employs reinforcement learning to optimize preferences over individual reasoning steps; second, a teacher-assisted variant, SFS-DPO-R, enhances the model’s self-verification and error-correction capabilities by incorporating explanatory rationales. Experimental results demonstrate that the proposed method significantly improves both the frequency and effectiveness of self-correction across multiple models and out-of-domain tasks, consistently outperforming existing baselines. These findings underscore the critical role of reinforcing step-level reasoning in enhancing the reliability of large language models.

Abstract

Achieving effective self-correction, where models verify and correct their own mistakes, remains a fundamental challenge for large language models (LLMs). In this work, we propose Self-Fix Step-DPO (SFS-DPO), a reinforcement learning based, two-stage framework for step-level self-verification and self-correction. The first stage strengthens step-level reasoning via step-level preference optimization, while the second stage explicitly trains models to self-verify and self-correct. We further introduce a teacher-assisted variant, SFS-DPO-R, which incorporates explanatory rationales for error verification to provide stronger corrective signals. Comprehensive in-domain and out-of-domain evaluations across multiple LLMs demonstrate that SFS-DPO and SFS-DPO-R consistently outperform prior step-level training baselines. Our analysis further reveals improvements in self-correction frequency and effectiveness, highlighting the importance of strengthening step-level reasoning for robust performance.

Problem

• self-correction
• large language models
• step-level reasoning
• self-verification
• reasoning errors

Innovation

• step-level reasoning
• self-correction
• preference optimization
• reinforcement learning
• explanatory rationales