How does Chain of Thought decompose complex tasks?

2026-04-10

AI Summary

This work addresses the high error rates in direct classification for complex language tasks with numerous categories. The authors propose decomposing such tasks into a tree-structured, multi-step classification process that emulates chain-of-thought (CoT) reasoning, where the number of classes per step (degree) and reasoning depth are jointly optimized to minimize overall error. Theoretical analysis reveals that classification error decays as a power law with respect to the degree and identifies a critical degree threshold: below this threshold, increasing depth degrades performance, whereas above it, an optimal depth exists that minimizes error. Furthermore, the study establishes a fundamental lower bound on error for any fixed degree and demonstrates that CoT-style reasoning with appropriate depth can closely approach this bound, substantially reducing prediction errors.

Technology Category

• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Many language tasks can be modeled as classification problems where a large language model (LLM) is given a prompt and selects one among many possible answers. We show that the classification error in such problems scales as a power law in the number of classes. This has a dramatic consequence: the prediction error can be reduced substantially by splitting the overall task into a sequence of smaller classification problems, each with the same number of classes ("degree"). This tree-structured decomposition models chain-of-thought (CoT). It has been observed that CoT-based predictors perform better when they"think'", i.e., when they develop a deeper tree, thus decomposing the problem into a larger number of steps. We identify a critical threshold for the degree, below which thinking is detrimental, and above which there exists an optimal depth that minimizes the error. It is impossible to surpass this minimal error by increasing the depth of thinking.

Problem

• Chain of Thought
• classification error
• task decomposition
• power law
• optimal depth

Innovation

• Chain of Thought
• classification error
• power law
• tree-structured decomposition
• optimal depth