Markov Chain of Thought for Efficient Mathematical Reasoning

2024-10-23

arXiv.org

AI Summary

Chain-of-thought (CoT) reasoning in mathematical problem solving suffers from excessive token consumption, high KV cache overhead, and low inference efficiency due to long dependency chains. Method: We propose Markovian Chain-of-Thought (MCoT), modeling each reasoning step as a text state augmented with executable Python code; an integrated code interpreter enables automatic verification and dynamic history compression, reducing redundant intermediate steps to equivalent problem representations. MCoT formally recasts multi-step CoT as a Markov process, eliminating reliance on full-history KV caching. We construct the MCoTInstruct dataset—grounded in symbolic reasoning, code execution, and instruction tuning—and adapt it to mainstream LLM inference pipelines. Results: Experiments show MCoT matches baseline accuracy on mathematical reasoning while significantly reducing latency (−38% on average) and KV cache memory usage (−62%), validating a novel paradigm for efficient long-horizon reasoning.

Technology Category

• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Machine Learning: Mixture of Experts (MoE)

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Chain of Thought (CoT) of multi-step benefits from the logical structure of the reasoning steps and task-specific actions, significantly enhancing the mathematical reasoning capabilities of large language models. As the prevalence of long CoT, the number of reasoning steps exceeds manageable token limits and leads to higher computational demands. Inspired by the fundamental logic of human cognition,"derive, then reduce", we conceptualize the standard multi-step CoT as a novel Markov Chain of Thought (MCoT). In this study, we consider the mathematical reasoning task, defining each reasoning step as text accompanied by a Python code snippet. To facilitate a longer reasoning path, self-correction is enabled through interactions with the code interpreter. Our MCoT aims to compress previous reasoning steps into a simplified question, enabling efficient next-step inference without relying on a lengthy KV cache. In our experiments, we curate the $ exttt{MCoTInstruct}$ dataset, and the empirical results indicate that MCoT not only significantly enhances efficiency but also maintains comparable accuracy. While much remains to be explored, this work paves the way for exploring the long CoT reasoning abilities of LLMs. The code is available at https://github.com/james-yw/Markov-Chain-of-Thought

Problem

• Addresses computational inefficiency in long Chain of Thought reasoning.
• Proposes Markov Chain of Thought to compress reasoning steps.
• Enhances mathematical reasoning in large language models.

Innovation

• Markov Chain of Thought for efficient reasoning
• Self-correction via code interpreter interaction
• Compresses steps for efficient next-step inference