Geometric Signatures of Reasoning: A Spectral Perspective on Task Hardness

2026-07-01

AI Summary

This work investigates the geometric structure of chain-of-thought reasoning trajectories in large language models and its relationship to task difficulty and answer correctness. Modeling the reasoning process as a discrete curve in the hidden state space, the study introduces an effective dimensionality measure, $d_\rho$, to quantify trajectory complexity and employs spectral analysis alongside geometric functionals derived from positional and kinematic properties to characterize these trajectories. The findings reveal that flatter eigenvalue spectra correspond to more difficult tasks, and that dynamical features extracted from merely the first 20% of generated tokens suffice to effectively predict final answer correctness. Evaluated on the MATH500 dataset, $d_\rho$ achieves an AUC of 0.93 in distinguishing between easy and hard problems, demonstrating both the efficacy and predictive power of the proposed approach.

Abstract

Chain-of-thought (CoT) reasoning enables large language models (LLMs) to solve complex problems by generating intermediate reasoning steps. While much attention has been paid to the length and content of these reasoning chains, far less is known about their internal geometry. We study the \emph{geometry} of CoT trajectories in the hidden state space of transformer models, formalizing each reasoning chain as a discrete curve in $\mathbb{R}^d$ and characterizing it through spectral, positional, and kinematic geometric functionals. We introduce the effective dimension $d_ρ$ as a measure of trajectory complexity and show theoretically that trajectories with flatter eigenvalue spectra correspond to harder tasks, as they explore more of the hidden dimensions. Lastly, we explore how kinematic features of the trajectory, mean position, positional dispersion, initial and current hidden states, mean velocity, mean speed, and speed dispersion, can be used to predict solution correctness before generation is complete, and may inform future early-stopping strategies. Experimentally, on mathematical reasoning problems from the MATH500 dataset, $d_ρ$ achieves $0.93$ AUC in distinguishing easy from hard problems, while kinematic features potentially can predict correctness from only the first $20\%$ of generated tokens. These correctness signatures transfer across questions of varying difficulty, establishing that the shape of a model's internal reasoning trajectory is a principled window into both task hardness and solution quality.

Problem

• chain-of-thought reasoning
• task hardness
• hidden state geometry
• trajectory complexity
• solution correctness

Innovation

• geometric reasoning
• chain-of-thought
• spectral analysis
• effective dimension
• kinematic features