Provable Benefits of RLVR over SFT for Reasoning Models: Learning to Backtrack Efficiently

2026-06-22

AI Summary

This work addresses the inefficiency of supervised fine-tuning (SFT) in acquiring effective backtracking capabilities for reasoning tasks, which leads to suboptimal computational efficiency. The authors formalize chain-of-thought reasoning as a path search problem over a graph and propose a reinforcement learning approach with verifiable rewards (RLVR). They theoretically demonstrate, for the first time, that RLVR can learn efficient backtracking from outcome-based rewards, whereas SFT fundamentally cannot, revealing an exponential gap in reasoning complexity between the two paradigms. Empirical results show that RLVR substantially enhances the model’s ability to identify challenging decision points and allocate computational resources accordingly. Furthermore, the learned backtracking strategy can be effectively transferred to base models through trajectory distillation.

Abstract

Recent advances in large language models (LLMs) have demonstrated that reinforcement fine-tuning of pretrained base models can lead to significant gains in reasoning performance at inference time. In this work, we theoretically analyze why reinforcement fine-tuning induces better reasoning ability than purely supervised fine-tuning (SFT) methods. We model chain-of-thought (CoT) reasoning as a pathfinding problem on graphs and compare the popular method of reinforcement learning with verifiable rewards (RLVR) against traditional SFT. We prove that SFT, when trained on golden shortest paths without negative examples, fails to learn how to efficiently backtrack. In contrast, an RLVR-trained model can learn how to efficiently backtrack from dead ends using only outcome reward. This leads to an exponential separation in inference-time compute between the two methods, and demonstrates that RLVR leads the model to learn the location of difficult decisions in a reasoning chain, ultimately allowing for better allocation of inference-time compute. Finally, we show that the reasoning traces of an RLVR model can be distilled to train a base model to backtrack efficiently as well.

Problem

• reinforcement learning
• supervised fine-tuning
• reasoning models
• backtracking
• chain-of-thought

Innovation

• RLVR
• backtracking
• reasoning models
• chain-of-thought
• reinforcement learning