How Do Answer Tokens Read Reasoning Traces? Self-Reading Patterns in Thinking LLMs for Quantitative Reasoning

2026-04-21

AI Summary

This study investigates how large language models leverage attention mechanisms to “self-read” their own reasoning traces during answer generation in quantitative reasoning tasks, thereby enhancing answer reliability. Through analysis of attention patterns from answer tokens to reasoning traces, the authors identify that correct reasoning exhibits focused attention on key semantic anchors with an earlier shift in reading focus—a beneficial self-reading behavior—whereas incorrect reasoning shows diffuse attention. Building on this insight, they propose a training-free Self-Reading Quality (SRQ) guidance mechanism that dynamically evaluates and intervenes in the reasoning process by integrating geometric and semantic metrics. Experiments demonstrate that this approach significantly improves accuracy across multiple quantitative reasoning benchmarks, offering the first evidence of an intrinsic link between self-reading attention patterns and reasoning correctness.

Technology Category

• Knowledge Representation and Reasoning: Qualitative Reasoning
• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Thinking LLMs produce reasoning traces before answering. Prior activation steering work mainly targets on shaping these traces. It remains less understood how answer tokens actually read and integrate the reasoning to produce reliable outcomes. Focusing on quantitative reasoning, we analyze the answer-to-reasoning attention and observe a benign self-reading pattern aligned with correctness, characterized by a forward drift of the reading focus along the reasoning trace and a persistent concentration on key semantic anchors, whereas incorrect solutions exhibit diffuse and irregular attention pattern. We interpret this as internal certainty during answer decoding, where the model commits to a viable solution branch and integrates key evidence. Following this, we propose a training-free steering method driven by Self-Reading Quality (SRQ) scores combining geometric metrics for process control with semantic metrics for content monitoring. SRQ selects data to build steering vectors that guide inference toward benign self-reading and away from uncertain and disorganized reading. Experiments show that our method yields consistent accuracy gains.

Problem

• answer tokens
• reasoning traces
• quantitative reasoning
• self-reading
• attention pattern

Innovation

• self-reading pattern
• attention analysis
• reasoning trace
• training-free steering
• quantitative reasoning