Analyzing and Improving Chain-of-Thought Monitorability Through Information Theory

2026-02-20

AI Summary

This work addresses the unreliability of Chain-of-Thought (CoT) monitors in detecting undesirable behaviors—such as test-time exploitation—often stemming from insufficient information extraction or poor approximation of the monitoring function. For the first time, it formalizes CoT monitorability from an information-theoretic perspective, establishing that non-zero mutual information between the CoT and the output is necessary but insufficient for effective monitoring. The study identifies two key error sources: information gaps and steering errors. To mitigate these, it proposes a novel label-free joint optimization framework that combines conditional mutual information maximization with oracle-guided reinforcement training to systematically enhance monitor performance. Experiments demonstrate that this approach significantly improves monitoring accuracy across diverse settings, effectively suppresses CoT degradation, and alleviates reward hacking even under imperfect reward signals.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Knowledge Representation and Reasoning: Nonmonotonic Reasoning
• Reasoning under Uncertainty: Causality

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Chain-of-thought (CoT) monitors are LLM-based systems that analyze reasoning traces to detect when outputs may exhibit attributes of interest, such as test-hacking behavior during code generation. In this paper, we use information-theoretic analysis to show that non-zero mutual information between CoT and output is a necessary but not sufficient condition for CoT monitorability. We identify two sources of approximation error that may undermine the performance of CoT monitors in practice: information gap, which measures the extent to which the monitor can extract the information available in CoT, and elicitation error, which measures the extent to which the monitor approximates the optimal monitoring function. We further demonstrate that CoT monitorability can be systematically improved through targeted training objectives. To this end, we propose two complementary approaches: (a) an oracle-based method that directly rewards the monitored model for producing CoTs that maximize monitor accuracy, and (b) a more practical, label-free approach that maximizes conditional mutual information between outputs and CoTs. Across multiple different environments, we show both methods significantly improve monitor accuracy while preventing CoT degeneration even when training against a monitor, thereby mitigating reward hacking when the task reward is imperfectly specified.

Problem

• Chain-of-Thought
• monitorability
• information theory
• mutual information
• reward hacking

Innovation

• Chain-of-Thought
• monitorability
• information theory
• mutual information
• reward hacking