🤖 AI Summary
This work proposes a training-free method to assess the validity of mathematical reasoning by interpreting the attention matrices of large language models as dynamic graph adjacency matrices and applying spectral graph analysis to extract interpretable features—such as the Fiedler value, high-frequency energy ratio, graph signal smoothness, and spectral entropy. Evaluated across seven mainstream models, the approach achieves accuracy rates of 85.0%–95.6% (Cohen’s d = 3.30), which improve to 93%–95% after calibration. Notably, it correctly identifies valid proofs erroneously rejected by formal verifiers and reveals a shift in discriminative signals within Mistral-7B—from high-frequency energy ratio toward signal smoothness—highlighting the critical influence of attention mechanism design on reasoning reliability.
📝 Abstract
We present a training-free method for detecting valid mathematical reasoning in large language models through spectral analysis of attention patterns. By treating attention matrices as adjacency matrices of dynamic graphs over tokens, we extract four interpretable spectral diagnostics, the Fiedler value (algebraic connectivity), high-frequency energy ratio (HFER), graph signal smoothness, and spectral entropy, that exhibit statistically significant differences between valid and invalid mathematical proofs. Experiments across seven transformer models from four independent architectural families (Meta Llama, Alibaba Qwen, Microsoft Phi, and Mistral AI) demonstrate that this spectral signature produces effect sizes up to Cohen's $d = 3.30$ ($p<10^{-116}$), enabling 85.0--95.6\% classification accuracy under rigorous evaluation, with calibrated thresholds reaching 93--95\% on the full dataset. The method requires no training data, fine-tuning, or learned classifiers: a single threshold on a spectral metric suffices for high accuracy. Through systematic label correction, we discover that the spectral method detects logical coherence rather than compiler acceptance, identifying mathematically valid proofs that formal verifiers reject due to technical failures. We further identify an architectural dependency: Mistral-7B's Sliding Window Attention shifts the discriminative signal from HFER to late-layer Smoothness ($d = 2.09$, $p_{\text{MW}} = 1.16 \times 10^{-48}$), revealing that attention mechanism design affects which spectral features capture reasoning validity. These findings establish spectral graph analysis as a principled framework for reasoning verification with immediate applications to hallucination detection and AI safety monitoring.