Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters

2026-07-15

AI Summary

This work addresses a critical flaw in evaluating large language models’ predictive capabilities: the conflation of genuine forecasting with mere recall due to inadvertent exposure to post-hoc information in training or retrieval corpora. To remedy this, the authors propose Hindcast, a framework that freezes publicly available Reddit data at specific historical timestamps, restricting model access exclusively to information predating each target event. Predictive performance is then rigorously assessed against outcomes from the Polymarket prediction market using a dual-scoring mechanism, thereby eliminating future information leakage. This approach establishes a reproducible, temporally coherent benchmark for fair evaluation. Empirical results demonstrate that retrieval enhances prediction accuracy only when substantive, pre-event discussions exist on Reddit; in contrast, when prior content consists solely of speculative remarks, retrieval actively degrades performance.

Abstract

Forecasters are evaluated by backtesting, which replays resolved questions and grades the probability the system would have assigned before the outcome was known. For LLMs, two channels leak the answer into this test. A model that retrieves can surface reports written after the event, turning forecasting into a lookup, and each new model is trained on data closer to the event, so a question that lay in the future for last year's models sits inside this year's training data. Either way, the test grades recall while claiming to grade foresight. We introduce Hindcast, which closes both leaks by grading a model as if it stood at a chosen past date $t_0$, before the outcome existed in either channel. Hindcast replays resolved Polymarket prediction markets against a frozen snapshot of public Reddit, lets the model read only posts written before $t_0$, and scores each forecast against both what happened and the market's own price at $t_0$, itself a human forecast made from the same past information. Because the cutoff is set per market and the snapshot never changes, the evaluation re-runs on new markets as models improve, without going stale. Once the leak is closed, retrieval still helps most models, but only where Reddit discussed the event beforehand. Where the archive carried only speculation, retrieval hurts.

Problem

• forecasting
• information leakage
• backtesting
• large language models
• evaluation

Innovation

• Hindcast
• prediction markets
• large language models
• backtesting
• temporal leakage