Measuring Judgment Quality in Natural-Language Explanations: Evidence from Forecasting Tournaments

2026-06-29

AI Summary

This study addresses the challenge of effectively evaluating the judgment quality of natural language explanations at scale. Leveraging over 55,000 prediction–rationale pairs from forecasting tournaments, the authors propose Explanation Quality Metrics (EQMs)—a scalable, automated evaluation framework that uses large language models to score explanations across 60 theoretically grounded reasoning patterns. For the first time, this approach extracts interpretable signals from natural language explanations that are significantly associated with judgment accuracy. Experimental results demonstrate that EQMs robustly predict accuracy at both forecaster and forecast levels, outperforming conventional text analysis methods and emerging as the strongest indicator for identifying low-performing forecasters. The findings are successfully replicated in an independent study, confirming the method’s strong cross-dataset generalizability.

Abstract

Decision-makers routinely rely on expert judgments accompanied by written explanations, yet explanation quality is difficult to measure at scale. Forecasting tournaments offer a natural testing ground: probabilistic judgments are paired with natural-language rationales and scored against realized outcomes. We introduce Explanation Quality Markers (EQMs), a set of sixty theory-guided reasoning patterns scored by large language models (LLMs). In a pre-registered analysis of over 55,000 forecast-rationale pairs from a multiyear forecasting tournament, EQMs predict accuracy at both the forecast and forecaster levels, consistently outperforming pre-LLM text-analysis methods. More than 90% of statistically significant pattern-level EQM-accuracy correlations match our directional hypotheses. The signal is asymmetric: EQMs identify likely underperformers more reliably than they distinguish the very best forecasters. Benchmarked against traditional indicators of forecasting skill, EQMs are the strongest predictor at the forecast level and competitive at the forecaster level, though weaker than prior accuracy. Human ratings of rationale quality are less consistently correlated with accuracy and place disproportionate weight on rationale length. Results transfer to an independent forecasting study. EQMs provide a scalable, interpretable method for extracting judgment-relevant information from written explanations.

Problem

• explanation quality
• judgment quality
• natural-language explanations
• forecasting tournaments
• reasoning patterns

Innovation

• Explanation Quality Markers
• large language models
• forecasting tournaments
• judgment quality
• natural-language explanations