🤖 AI Summary
Current AI evaluation lacks large-scale, diverse, and verifiable predictive questions, limiting the assessment of general intelligence. This work proposes the first end-to-end automated framework that leverages large language model (LLM)-driven web research agents to generate and resolve high-quality, real-world forecasting questions at scale. By overcoming the constraints of manual annotation and fixed data sources, the approach enables iterative refinement of forecasting strategies. Among the 1,499 generated questions, 96% are clearly verifiable—surpassing Metaculus in this regard—and achieve a resolution accuracy of 95%. Experiments demonstrate that stronger LLMs yield better performance, and that decomposing complex questions significantly improves forecasting accuracy, reducing the Brier score from 0.141 to 0.132.
📝 Abstract
Forecasting future events is highly valuable in decision-making and is a robust measure of general intelligence. As forecasting is probabilistic, developing and evaluating AI forecasters requires generating large numbers of diverse and difficult questions, and accurately resolving them. Previous efforts to automate this laborious work relied on recurring data sources (e.g., weather, stocks), limiting diversity and utility. In this work, we present a system for generating and resolving high-quality forecasting questions automatically and at scale using LLM-powered web research agents. We use this system to generate 1499 diverse, real-world forecasting questions, and to resolve them several months later. We estimate that our system produces verifiable, unambiguous questions approximately 96% of the time, exceeding the rate of Metaculus, a leading human-curated forecasting platform. We also find that our system resolves questions at approximately 95% accuracy. We verify that forecasting agents powered by more intelligent LLMs perform better on these questions (Brier score of 0.134 for Gemini 3 Pro, 0.149 for GPT-5, and 0.179 for Gemini 2.5 Flash). Finally, we demonstrate how our system can be leveraged to directly improve forecasting, by evaluating a question decomposition strategy on a generated question set, yielding a significant improvement in Brier scores (0.132 vs. 0.141).