When Can Model-Free Reinforcement Learning be Enough for Thinking?

2025-06-20

AI Summary

This work investigates the emergence mechanism of “thinking” behaviors in model-free reinforcement learning (RL). Method: We propose the Thought-Markov Decision Process (Thought-MDP) framework—a formal theoretical model that first defines endogenous thinking actions as online policy improvement steps and rigorously proves their equivalence to standard RL updates. We identify policy initialization as the critical condition for thinking emergence and derive a general necessary and sufficient criterion characterizing when model-free RL acquires reasoning capability. Results: Through theoretical modeling, formal proofs, and empirical analysis on open-source large language models (LLMs), we validate that mainstream LLMs satisfy the theoretical predictions. On synthetic reasoning tasks, explicitly incorporating thinking actions significantly improves data efficiency. This work establishes the first empirically testable theoretical foundation for understanding the RL origins of LLM reasoning capabilities.

Technology Category

• Machine Learning: Reinforcement Learning
• Knowledge Representation and Reasoning: Reasoning with Beliefs
• Reasoning under Uncertainty: Sequential Decision Making

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Recent work on large language models has demonstrated the use of model-free reinforcement learning (RL) to train reasoning-like capabilities. The emergence of"thinking"through model-free RL is interesting as thinking actions neither produce reward nor change the external world state to one where the agent is more likely to get reward. This paper seeks to build a domain-independent understanding of when model-free RL will lead to"thinking"as a strategy for reward maximization. To build this understanding, we first introduce a theoretical model which we call a extit{thought Markov decision process} (MDP). Thought MDPs minimally extend the classical MDP model to include an abstract notion of thought state and thought action. Using the thought MDP model, we prove the importance of policy initialization in determining whether or not thinking emerges and show formally that thought actions are equivalent to the agent choosing to perform a step of policy improvement before continuing to act. We then show that open-source LLMs satisfy the conditions that our theory predicts are necessary for model-free RL to produce thinking-like behavior. Finally, we hypothesize sufficient conditions that would enable thinking to be learned outside of language generation and introduce a toy domain where a combination of multi-task pre-training and designated thought actions enable more data-efficient RL compared to non-thinking agents.

Problem

• When does model-free RL enable thinking-like behavior
• How policy initialization affects emergence of thinking
• Sufficient conditions for learning thinking outside language

Innovation

• Introduces thought MDPs for abstract reasoning
• Proves policy initialization's role in thinking
• Validates conditions with open-source LLMs