A Survey of Slow Thinking-based Reasoning LLMs using Reinforced Learning and Inference-time Scaling Law

2025-05-05

AI Summary

Large language models (LLMs) exhibit insufficient deep reasoning capabilities on complex tasks such as mathematical reasoning, visual understanding, medical diagnosis, and multi-agent debate. Method: This work is the first to systematically integrate Kahneman’s “slow thinking” cognitive theory into LLM architecture, proposing three innovations: (1) test-time dynamic compute scaling, (2) strategy-driven self-evolving reinforcement learning, and (3) a slow-thinking framework combining long-chain and hierarchical collaboration. Technically, it unifies dynamic verification, strategy network modeling, reward-guided sampling, and an enhanced chain-of-thought (CoT) mechanism to jointly optimize reasoning depth and computational efficiency. Contribution/Results: Based on a systematic review of 100+ studies, we construct a unified technical taxonomy and demonstrate significant improvements in reasoning accuracy and robustness across multiple benchmarks, establishing a novel paradigm for human-like, scalable, and trustworthy scientific reasoning and decision support in LLMs.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

This survey explores recent advancements in reasoning large language models (LLMs) designed to mimic"slow thinking"- a reasoning process inspired by human cognition, as described in Kahneman's Thinking, Fast and Slow. These models, like OpenAI's o1, focus on scaling computational resources dynamically during complex tasks, such as math reasoning, visual reasoning, medical diagnosis, and multi-agent debates. We present the development of reasoning LLMs and list their key technologies. By synthesizing over 100 studies, it charts a path toward LLMs that combine human-like deep thinking with scalable efficiency for reasoning. The review breaks down methods into three categories: (1) test-time scaling dynamically adjusts computation based on task complexity via search and sampling, dynamic verification; (2) reinforced learning refines decision-making through iterative improvement leveraging policy networks, reward models, and self-evolution strategies; and (3) slow-thinking frameworks (e.g., long CoT, hierarchical processes) that structure problem-solving with manageable steps. The survey highlights the challenges and further directions of this domain. Understanding and advancing the reasoning abilities of LLMs is crucial for unlocking their full potential in real-world applications, from scientific discovery to decision support systems.

Problem

• Enhancing reasoning LLMs via dynamic computation scaling for complex tasks
• Improving decision-making in LLMs using reinforced learning strategies
• Developing slow-thinking frameworks to structure problem-solving steps

Innovation

• Dynamic computation scaling at inference time
• Reinforced learning for iterative decision refinement
• Slow-thinking frameworks structuring problem-solving steps