KisMATH: Do LLMs Have Knowledge of Implicit Structures in Mathematical Reasoning?

2025-07-15

AI Summary

This work investigates whether large language models (LLMs) implicitly encode structured causal dependencies underlying mathematical reasoning, and how chain-of-thought (CoT) prompting supports this capability. Method: We propose the Causal Chain Graph (CCG)—a directed acyclic graph that models fine-grained causal mediation and path preferences among reasoning steps—and automatically construct it from model-generated reasoning traces. To enable rigorous analysis, we introduce KisMATH, a benchmark comprising 1,671 math problems annotated with step-level causal structure. Contribution/Results: Through multi-model empirical evaluation across 15 open-source LLMs and graph-alignment interventions, we demonstrate that LLMs consistently follow CCG-structured causal paths during reasoning. This provides the first causally grounded, graph-based interpretability evidence for CoT’s internal mechanism, revealing that mathematical reasoning in LLMs is not merely sequential but governed by latent causal structure—thereby advancing our understanding of the fundamental nature of LLM-based mathematical reasoning.

Technology Category

• Reasoning under Uncertainty: Causality
• Machine Learning: Causal Learning
• Knowledge Representation and Reasoning: Action, Change, and Causality

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Chain-of-thought traces have been shown to improve performance of large language models in a plethora of reasoning tasks, yet there is no consensus on the mechanism through which this performance boost is achieved. To shed more light on this, we introduce Causal CoT Graphs (CCGs), which are directed acyclic graphs automatically extracted from reasoning traces that model fine-grained causal dependencies in the language model output. A collection of $1671$ mathematical reasoning problems from MATH500, GSM8K and AIME, and their associated CCGs are compiled into our dataset -- extbf{KisMATH}. Our detailed empirical analysis with 15 open-weight LLMs shows that (i) reasoning nodes in the CCG are mediators for the final answer, a condition necessary for reasoning; and (ii) LLMs emphasise reasoning paths given by the CCG, indicating that models internally realise structures akin to our graphs. KisMATH enables controlled, graph-aligned interventions and opens up avenues for further investigation into the role of chain-of-thought in LLM reasoning.

Problem

• Mechanism of performance boost in LLM reasoning
• Modeling causal dependencies in reasoning traces
• LLM internal realization of reasoning structures

Innovation

• Introduces Causal CoT Graphs (CCGs)
• Automatically extracts reasoning dependencies
• Enables controlled graph-aligned interventions