🤖 AI Summary
This work addresses the challenge of hallucination in large language models, which often produce plausible yet incorrect answers during reasoning, thereby undermining reliability. The authors propose a novel approach that integrates Direct Preference Optimization (DPO) with Low-Rank Adaptation (LoRA) to simultaneously optimize forward chain-of-thought reasoning and backward verification objectives. Their analysis reveals a fundamental trade-off between these two strategies: forward training improves problem-solving accuracy—raising performance on GSM8K from 83.1% to 86.6%—while backward training substantially reduces the false positive rate, decreasing it from 13.4% to 4.3%. These complementary signals jointly enhance the model’s calibration and output confidence, leading to more trustworthy and self-consistent reasoning.
📝 Abstract
Large language models exhibit impressive reasoning capabilities yet frequently generate plausible but incorrect solutions, a phenomenon commonly termed hallucination. This paper investigates the effect of training objective composition on reasoning reliability through Direct Preference Optimization. Two complementary training signals are examined: forward chain-of-thought generation, which trains the model to produce correct reasoning traces, and backward verification, which trains the model to verify and acknowledge errors in candidate solutions. Experiments on GSM8K reveal a fundamental trade-off between these objectives. Forward-only DPO training achieves the highest accuracy improvement, increasing from 83.1% to 86.6% (+3.5 percentage points), while backward-only training yields minimal accuracy gains but substantially reduces the false positive rate from 13.4% to 4.3%. Notably, both training variants reduce acknowledgement rate compared to the baseline, suggesting that preference optimization increases model confidence in its outputs. These findings indicate that forward and backward reasoning objectives provide distinct and complementary learning signals: forward training improves problem-solving capability, while backward training improves verification calibration. The complete training and evaluation pipeline, implemented efficiently through Low-Rank Adaptation, is released to facilitate further research.