Prompt Engineering Large Language Models' Forecasting Capabilities

2025-06-02

AI Summary

This study investigates whether prompt engineering—without fine-tuning or external tools—can enhance large language models’ (LLMs) performance on complex time-series and probabilistic forecasting tasks. Method: We systematically evaluate 38 prompting strategies—including single-step, composite, and cross-source prompts—across state-of-the-art models (Claude 3.5, GPT-4o, Llama 3.1 405B, and o1 series), with particular focus on advanced techniques such as chain-of-thought reasoning, probability calibration, and Bayesian-inspired heuristics. Contribution/Results: Contrary to prevailing assumptions, most advanced prompting strategies degrade forecasting accuracy significantly; only baseline-rate prompting yields consistent, albeit marginal, improvements. This work provides the first empirical evidence of fundamental limitations in widely adopted prompting paradigms for predictive tasks, demonstrating that prompt optimization alone cannot overcome LLMs’ intrinsic constraints in uncertainty modeling. Our findings offer critical, evidence-based guidance for paradigm selection in LLM-driven forecasting applications.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large language model performance can be improved in a large number of ways. Many such techniques, like fine-tuning or advanced tool usage, are time-intensive and expensive. Although prompt engineering is significantly cheaper and often works for simpler tasks, it remains unclear whether prompt engineering suffices for more complex domains like forecasting. Here we show that small prompt modifications rarely boost forecasting accuracy beyond a minimal baseline. In our first study, we tested 38 prompts across Claude 3.5 Sonnet, Claude 3.5 Haiku, GPT-4o, and Llama 3.1 405B. In our second, we introduced compound prompts and prompts from external sources, also including the reasoning models o1 and o1-mini. Our results show that most prompts lead to negligible gains, although references to base rates yield slight benefits. Surprisingly, some strategies showed strong negative effects on accuracy: especially encouraging the model to engage in Bayesian reasoning. These results suggest that, in the context of complex tasks like forecasting, basic prompt refinements alone offer limited gains, implying that more robust or specialized techniques may be required for substantial performance improvements in AI forecasting.

Problem

• Assessing prompt engineering's impact on LLM forecasting accuracy
• Evaluating minimal gains from prompt modifications in forecasting
• Identifying negative effects of certain reasoning strategies on accuracy

Innovation

• Tested 38 prompts across multiple LLMs
• Introduced compound and external prompts
• Found basic prompt refinements offer limited gains