Where Do CoT Training Gains Land in LLM based Agents?

2026-06-25

AI Summary

This study investigates whether chain-of-thought (CoT) training in large language model agents genuinely enhances reasoning capabilities or merely improves the ability to directly predict actions from prompts. Through comparative action prediction analyses, evaluation of training checkpoints, and selective masking of action tokens, the authors find that the primary benefit of CoT training stems from improved prompt-action alignment rather than strengthened internal reasoning. Moreover, as training progresses, models increasingly rely less on CoT for action correction. Building on these insights, the work proposes an intervention strategy that selectively masks action supervision signals during training, which effectively boosts out-of-distribution generalization performance.

Abstract

Chain-of-thought (CoT) reasoning is widely used in language-model agents, but prior work has shown that verbalized CoT is not always faithful and may instead reflect post-hoc reasoning, which means the model already knows the answer before reasoning. We therefore ask what CoT training is actually improving: is the model getting better at changing its action through generated reasoning, or is it getting better at predicting the action directly from the prompt? We study this question by comparing \emph{prompt actions} (predicting action without CoT) with CoT actions (predicting action with CoT). Across checkpoints, prompt-action quality improves substantially. While interacting with the environment, the relative advantage of CoT actions over prompt actions remains similar, showing that CoT training does not widen the advantage of CoT reasoning, and it helps to improve the quality of prompt actions. We further find that later checkpoints are less likely to revise the action in response to CoT, suggesting greater reliance on the prompt. Motivated by these patterns, we selectively mask action-token supervision on a fraction of training examples. This intervention improves out-of-domain generalization.

Problem

• Chain-of-Thought
• language-model agents
• prompt actions
• reasoning faithfulness
• action prediction

Innovation

• Chain-of-Thought
• prompt actions
• action-token masking
• out-of-domain generalization
• LLM agents