What LLM Forecasters Know but Don't Say: Probing Internal Representations for Calibration and Faithfulness

2026-07-08

AI Summary

Although large language models achieve high prediction accuracy, they often exhibit poor calibration and their chain-of-thought reasoning may not faithfully reflect the true basis of their predictions. This work proposes a representation-based probing method that pools intermediate-layer activations to uncover the model’s internal decision rationale. It demonstrates, for the first time, that internal representations provide a more reliable indicator of prediction grounds than chain-of-thought outputs, serving effectively as both a calibration tool and a “lie detector.” By integrating evidence ablation, perturbation injection, and pre-inference answer distribution analysis, the approach substantially improves calibration performance, correctly predicting the direction of perturbation effects in 84% of cases and reducing generated tokens by 30–47% through answer-aware routing without compromising accuracy.

Abstract

Large language models fine-tuned for forecasting can be accurate yet poorly calibrated, and their chain-of-thought (CoT) reasoning may not faithfully reflect the evidence behind a forecast. We ask whether internal representations offer a more direct window into both. Working with Eternis-Forecaster 8B on OpenForesight, we train representation-pooling probes on intermediate activations and find they achieve substantially better calibration; a result that also holds for GLM-4.7-Flash and GLM-4.5-Air. We then assess CoT faithfulness through evidence ablation and diversionary injection: removing an influential source in the prompt often changes the model's forecast while leaving the reasoning trace untouched. The same probes function as lie detectors: their activations track behavioral shifts far better than the reasoning trace does, and they also predict the direction of change in 84% of cases, including when the CoT conceals the perturbation's influence. Finally, forced answering reveals that forecasts are largely fixed before reasoning begins: a single pre-reasoning pass recovers the committed answer and confidence, and routing questions by the spread of this pre-set answer distribution saves 30-47% of generated tokens, with no loss of accuracy. Together, these results establish probing internal representations as a practical tool for calibrating, auditing, and triaging language model forecasters and reasoning models more broadly.

Problem

• calibration
• faithfulness
• chain-of-thought
• forecasting
• internal representations

Innovation

• internal representations
• calibration
• chain-of-thought faithfulness
• probing
• forecasting