ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities

2024-09-30

arXiv.org

AI Summary

Existing forecasting evaluation frameworks lack dynamic, future-oriented benchmarks that isolate models from temporal leakage and adapt to evolving real-world events. Method: We introduce ForecastBench—the first fully time-isolated, dynamic benchmark for AI forecasting capability—automatically generating and continuously updating 1,000 future-event prediction questions with zero answer leakage risk. Rigorous evaluation employs timestamp validation, multi-source human forecasts, and strict statistical testing (p < 0.01) to assess experts, the general public, and large language models (LLMs). Contribution/Results: On a 200-question subset, domain experts significantly outperform state-of-the-art LLMs (p < 0.01), exposing fundamental limitations of current LLMs in genuine forward-looking prediction tasks. ForecastBench establishes a scalable, reproducible, and openly accessible evaluation infrastructure (forecastbench.org), providing the first standardized, real-time platform for advancing research on predictive intelligence.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Forecasts of future events are essential inputs into informed decision-making. Machine learning (ML) systems have the potential to deliver forecasts at scale, but there is no framework for evaluating the accuracy of ML systems on a standardized set of forecasting questions. To address this gap, we introduce ForecastBench: a dynamic benchmark that evaluates the accuracy of ML systems on an automatically generated and regularly updated set of 1,000 forecasting questions. To avoid any possibility of data leakage, ForecastBench is comprised solely of questions about future events that have no known answer at the time of submission. We quantify the capabilities of current ML systems by collecting forecasts from expert (human) forecasters, the general public, and LLMs on a random subset of questions from the benchmark ($N=200$). While LLMs have achieved super-human performance on many benchmarks, they perform less well here: expert forecasters outperform the top-performing LLM (p-value $<0.01$). We display system and human scores in a public leaderboard at www.forecastbench.org.

Problem

• Artificial Intelligence
• Human Prediction
• Benchmark Platform

Innovation

• ForecastBench
• AI Prediction Capabilities
• Expert vs AI Forecasting