Thinker: Training LLMs in Hierarchical Thinking for Deep Search via Multi-Turn Interaction

2025-11-11

AI Summary

Existing end-to-end reinforcement learning methods for LLMs invoking external retrievers to solve complex problems lack explicit supervision over the reasoning process, resulting in insufficient logical coherence and rigor. This paper proposes a supervisable and verifiable hierarchical reasoning framework: it decomposes problems into dependency-ordered subproblems via multi-turn interaction, integrates dual-modal representations—natural language and logical functions—and introduces a knowledge boundary determination mechanism to suppress redundant retrieval while explicitly modeling inter-subproblem dependencies. The framework supports hybrid retrieval from both knowledge bases and the web, substantially enhancing reasoning controllability and interpretability. It achieves baseline performance with only hundreds of training samples; after full-scale training, it consistently outperforms state-of-the-art methods across multiple datasets and LLMs of varying scales.

Technology Category

• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Efficient retrieval of external knowledge bases and web pages is crucial for enhancing the reasoning abilities of LLMs. Previous works on training LLMs to leverage external retrievers for solving complex problems have predominantly employed end-to-end reinforcement learning. However, these approaches neglect supervision over the reasoning process, making it difficult to guarantee logical coherence and rigor. To address these limitations, we propose Thinker, a hierarchical thinking model for deep search through multi-turn interaction, making the reasoning process supervisable and verifiable. It decomposes complex problems into independently solvable sub-problems, each dually represented in both natural language and an equivalent logical function to support knowledge base and web searches. Concurrently, dependencies between sub-problems are passed as parameters via these logical functions, enhancing the logical coherence of the problem-solving process. To avoid unnecessary external searches, we perform knowledge boundary determination to check if a sub-problem is within the LLM's intrinsic knowledge, allowing it to answer directly. Experimental results indicate that with as few as several hundred training samples, the performance of Thinker is competitive with established baselines. Furthermore, when scaled to the full training set, Thinker significantly outperforms these methods across various datasets and model sizes. The source code is available at https://github.com/OpenSPG/KAG-Thinker.

Problem

• Enhancing LLM reasoning through hierarchical decomposition of complex problems
• Ensuring logical coherence via dual representation of sub-problems
• Optimizing external searches by determining LLM knowledge boundaries

Innovation

• Hierarchical thinking model for deep search
• Dual representation of sub-problems in language and logic
• Knowledge boundary determination to avoid unnecessary searches