Hierarchical Chain-of-Thought Prompting: Enhancing LLM Reasoning Performance and Efficiency

2026-03-31

AI Summary

This work addresses the inefficiency and logical fragmentation often observed in traditional Chain-of-Thought (CoT) prompting during complex multi-step reasoning, which frequently arises from redundant intermediate steps. To overcome these limitations, the authors propose Hierarchical Chain-of-Thought (Hi-CoT), a novel approach that introduces a structured, hierarchical reasoning paradigm. Hi-CoT alternates between high-level directive planning and low-level step-by-step execution, thereby decomposing intricate tasks into logically coherent sub-steps. Empirical evaluations demonstrate that this method substantially enhances both accuracy and efficiency in long-horizon reasoning for large language models. Across multiple mainstream models and mathematical reasoning benchmarks, Hi-CoT achieves an average accuracy improvement of 6.2%—reaching up to 61.4% in certain cases—while simultaneously reducing reasoning trajectory length by 13.9%, underscoring the critical role of a strict hierarchical structure in boosting performance.

Abstract

Chain-of-Thought (CoT) prompting has significantly improved the reasoning capabilities of large language models (LLMs). However, conventional CoT often relies on unstructured, flat reasoning chains that suffer from redundancy and suboptimal performance. In this work, we introduce Hierarchical Chain-of-Thought (Hi-CoT) prompting, a structured reasoning paradigm specifically designed to address the challenges of complex, multi-step reasoning. Hi-CoT decomposes the reasoning process into hierarchical substeps by alternating between instructional planning and step-by-step execution. This decomposition enables LLMs to better manage long reasoning horizons and maintain logical coherence. Extensive evaluations across diverse LLMs and mathematical reasoning benchmarks show that Hi-CoT consistently improves average accuracy by 6.2% (up to 61.4% on certain models and tasks) while reducing reasoning trace length by 13.9% compared to CoT prompting. We further show that accuracy and efficiency are maximized when models strictly adhere to the hierarchical structure. Our code is available at https://github.com/XingshuaiHuang/Hi-CoT.

Problem

• Chain-of-Thought
• reasoning
• large language models
• hierarchical reasoning
• multi-step reasoning

Innovation

• Hierarchical Chain-of-Thought
• structured reasoning
• instructional planning
• reasoning efficiency
• large language models