Knowledge Knows, Verbalization Tells: Disentangling Latent Directions for Mathematical Solvability in LLMs

2026-07-06

AI Summary

This work addresses the tendency of large language models (LLMs) to generate hallucinated solutions to unsolvable mathematical problems, often stemming from a conflation between internal knowledge and surface-level linguistic expression. The study presents the first representation-level disentanglement of solvability knowledge from linguistic output, demonstrating that fabricated answers primarily arise from expressive biases rather than factual inaccuracies in stored knowledge. Leveraging techniques such as representational probing, activation steering, and unsolvable prompts, the authors identify distinct, linearly decodable directions in the hidden state space corresponding to solvability judgment and verbal response generation across multiple mainstream LLMs. By selectively modulating the expressive pathway, the approach significantly reduces hallucination rates and enhances the model’s ability to correctly abstain from answering unsolvable questions.

Abstract

Although LLMs have made significant progress in mathematical reasoning, determining whether a mathematical problem is solvable remains a fundamental yet challenging capability. While recent studies have probed internal representations of model solvability beliefs, verbalization has primarily been studied behaviorally rather than as an internal representation, limiting its analysis and manipulation. We address this gap by separately probing representations of solvability knowledge and verbalization, allowing us to disentangle the two within model hidden states. Across multiple LLMs, we show that knowledge and verbalization are encoded as distinct, linearly decodable representations and that fabrication is primarily associated with changes in verbalization rather than the underlying knowledge. Prompting with unsolvability cues reduces fabrication primarily by shifting verbalization, while activation steering demonstrates that these representations can be echanistically manipulated to improve model abstention.

Problem

• mathematical solvability
• large language models
• verbalization
• knowledge representation
• hallucination

Innovation

• disentanglement
• solvability
• verbalization
• activation steering
• fabrication