Leveraging Log Probabilities in Language Models to Forecast Future Events

2025-01-08

AI Summary

To address insufficient accuracy in text-driven future event prediction, this paper proposes a multi-step probabilistic estimation framework leveraging log-probability calibration of large language models (LLMs). It is the first work to systematically transform internal LLM log probabilities into calibrated event occurrence probabilities and integrate trend-evolution trajectory modeling for probabilistic foresight across 15 event types. The method significantly improves prediction reliability and interpretability, achieving a Brier score of 0.186—26% better than random baseline and 19% superior to leading AI systems. Key contributions are: (1) a novel calibration mechanism converting LLM log probabilities into well-calibrated event probabilities; (2) a trend-aware multi-step probabilistic reasoning framework; and (3) empirical validation of the effectiveness and practicality of probabilistic reasoning for open-domain event forecasting.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Relational Probabilistic Models
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

In the constantly changing field of data-driven decision making, accurately predicting future events is crucial for strategic planning in various sectors. The emergence of Large Language Models (LLMs) marks a significant advancement in this area, offering advanced tools that utilise extensive text data for prediction. In this industry paper, we introduce a novel method for AI-driven foresight using LLMs. Building on top of previous research, we employ data on current trends and their trajectories for generating forecasts on 15 different topics. Subsequently, we estimate their probabilities via a multi-step approach based on log probabilities. We show we achieve a Brier score of 0.186, meaning a +26% improvement over random chance and a +19% improvement over widely-available AI systems.

Problem

• Future Event Prediction
• Text Data Analysis
• Decision Making

Innovation

• Large Language Models
• Log Probability Prediction
• Data-Assisted Decision Making