🤖 AI Summary
Recursive self-training is prone to recursive drift, mode collapse, and performance degradation due to the accumulation of intermediate reasoning errors. This work proposes the Neural-Symbolic Recursive Self-Alignment (NSRSA) framework, which introduces a symbolic verification mechanism at the level of individual reasoning steps for the first time. By integrating SymPy for arithmetic validation, logical consistency checking, and enforcement of domain-specific constraints, NSRSA effectively filters out samples with correct answers but flawed reasoning processes. Leveraging preference pairs constructed via Direct Preference Optimization (DPO), NSRSA rejects 34% of erroneous reasoning samples over five rounds of self-training on GSM8K using the Qwen3-4B-Thinking model, improving the model’s accuracy in judging reasoning correctness from 46% to 63%. This significantly enhances the reliability and stability of recursive self-training.
📝 Abstract
Recursive self-improvement--where a model iteratively trains on its own outputs--promises sustained capability growth but faces a fundamental obstacle: recursive drift. As models train on self-generated data across multiple iterations, errors in intermediate reasoning compound, leading to mode collapse and performance degradation. We propose Neuro-Symbolic Recursive Self-Alignment (NSRSA), which stabilizes iterative self-training by embedding a symbolic verification subsystem that gates training data quality at the reasoning step level. Unlike outcome-only filtering (which admits "lucky guesses" with flawed reasoning), NSRSA verifies each arithmetic operation via sympy, checks logical flow consistency across reasoning steps, and enforces domain constraints. We evaluate NSRSA on GSM8K using Qwen3-4B-Thinking across 5 self-training iterations under five conditions: no verification, outcome verification, majority voting, full NSRSA symbolic verification, and NSRSA with DPO. Our filtering analysis shows that NSRSA rejects approximately 34% of correct-answer solutions that pass outcome verification, eliminating "lucky guesses" with flawed reasoning from the training set. We further demonstrate that constructing DPO preference pairs from NSRSA verification teaches the model to distinguish sound from flawed reasoning (reward accuracy 46% to 63%). NSRSA provides an extensible framework that demonstrates how external symbolic verification can make recursive self-improvement measurable and reliable within domains where automated verification is available.