probabilistic forecast evaluation

This study addresses the limited interpretability of the Brier score in diagnosing deficiencies in probabilistic forecasts by proposing an algebraic rearrangement based on Yates’ covariance decomposition. The method cleanly decomposes the Brier score into three non-negative components: variance mismatch, insufficient correlation, and overall calibration bias. This decomposition is not only mathematically concise but also highly interpretable, explicitly revealing that perfect prediction requires simultaneous satisfaction of three conditions: matched variances, perfect positive correlation, and agreement in means. By elucidating the distinct sources of forecast error, the approach substantially enhances the diagnostic capability for evaluating probabilistic predictions and provides both a theoretical foundation and a practical tool for improving predictive models.

This study addresses the limitation of existing posterior calibration evaluation practices, which predominantly focus on aggregate performance while overlooking robustness across varying operational conditions within datasets. The authors propose the first conditionally stratified evaluation framework, employing preregistered experiments to systematically compare the robustness of temperature scaling (TEMP) and isotonic regression (ISO) under four controlled conditions. The assessment encompasses multiple dimensions—including calibration slope, Brier score, discrimination, and AUROC—and applies Holm’s correction for multiplicity in rigorous hypothesis testing. Results demonstrate that TEMP consistently achieves calibration slopes closer to one and superior, more stable Brier scores across all conditions; differences in discrimination between the two methods are negligible; and AUROC performance varies by condition, revealing that the robustness of calibration methods is highly dependent on both specific operational contexts and the choice of evaluation metric.

This study addresses the unification of calibration concepts across classification and regression tasks, aiming to ensure consistency between predicted distributions and observed outcomes for diverse data types—continuous, discrete, nominal, and binary. The work introduces modal calibration for nominal outcomes and establishes a hierarchical framework distinguishing full, partial, and average calibration. It proposes a generalized definition of calibration based on predictive distribution functionals—such as means, quantiles, and event probabilities—and leverages probability integral transforms alongside constructive algorithms for analysis. Key contributions include demonstrating the logical independence between dual probability integral transform (PIT) calibration and existing discrete calibration notions, clarifying implication and independence relationships among various calibration types, and providing reproducible methods for generating illustrative examples and counterexamples.

This study addresses the limitation of the traditional Brier score, which conflates calibration and discrimination in probabilistic forecasting, thereby hindering targeted optimization. The authors propose the Manokhin probability matrix, which for the first time decouples predictive quality into two orthogonal dimensions—calibration and discrimination—by constructing a two-dimensional diagnostic framework based on the Spiegelhalter Z-statistic and the expected rank interpretation of AUC-ROC. This framework categorizes classifiers into four archetypes: Eagle, Bull, Sloth, and Mole, and reveals a theoretical asymmetry: discrimination is inherently difficult to improve, whereas calibration can be effectively post-processed. Consequently, the paper advocates a practical guideline of “optimize discrimination first, then calibrate.” Large-scale evaluation on the TabArena-v0.1 benchmark across 21 classifiers and 5 calibrators shows that Venn-Abers calibration reduces log-loss by 6.5–12.6% for Bull-type models but slightly degrades Eagle-type performance, confirming an inherent trade-off between calibration and discrimination.

Classic Brier score neglects clinical decision impact, limiting its ability to assess the real-world utility of risk prediction models. To address this, we propose a clinical-utility-oriented weighted Brier score framework that integrates decision-sensitive weights to jointly quantify predictive accuracy and cost–benefit trade-offs. Our work is the first to decompose this score into discriminative and calibration components and to establish a theoretical connection with the H-measure, thereby bridging the gap between conventional calibration assessment and decision-theoretic evaluation. Leveraging decision modeling, weighted generalization, decomposition analysis, and rigorous theoretical derivation, we validate the framework on the Prostate Active Surveillance Study (PASS) cohort. Results demonstrate that the proposed score exhibits high sensitivity to clinically relevant risk thresholds and significantly outperforms both the classic Brier score and AUC. It thus serves as a comprehensive pre-deployment evaluation metric for clinical risk models.

Current performance evaluation metrics—such as accuracy and F1 score—are typically reported as point estimates, ignoring the uncertainty induced by data clustering structures. This oversight often leads to underestimation of variability and potentially misleading model comparisons. To address this, this work proposes a unified framework that expresses a broad class of performance metrics as smooth functionals of the confusion matrix probabilities. By integrating a cluster-robust sandwich variance estimator, the framework enables valid confidence interval construction, hypothesis testing, and paired model comparison. It represents the first systematic application of cluster-robust inference to predictive performance evaluation, accommodating both binary and multiclass settings, and further provides asymptotic theory–based methods for power and sample size calculations. Simulations demonstrate that the proposed approach achieves near-nominal coverage across diverse dependence structures and substantially outperforms conventional methods that ignore clustering; real-data analyses confirm that accounting for clustering can materially alter evaluation conclusions.

This work addresses the limitation of existing safety-critical systems, which typically evaluate only predictive accuracy while lacking rigorous validation of the overall calibration of predicted probability distributions. To bridge this gap, the authors propose a modular calibration testing framework that decouples the calibration process into four interchangeable components: data model, scoring rule, hypothesis formulation, and statistical test procedure. Built upon formal statistical hypothesis testing, the framework provides a single accept/reject decision for the entire predictive distribution. Crucially, it rejects only overly confident predictions while tolerating reasonable deviations, thereby balancing practicality with flexibility. Empirical evaluations on weather forecasting and robotic pose estimation tasks demonstrate that the framework effectively supports reliable deployment in safety-critical applications.

This work addresses the limitations of the standard Expected Calibration Error (ECE), which struggles to effectively capture overconfidence risks at high confidence levels and fails to evaluate the discriminative power of confidence scores with respect to prediction correctness. To overcome these issues, the authors propose the Calibrated Size Ratio (CSR) as a more sensitive calibration metric and introduce the risk probability \(P_{\text{risk}}\) to quantify overconfidence. Furthermore, they systematically extend confidence-weighting mechanisms to various classification metrics for the first time, yielding novel measures such as cwA and cwAUC to assess the discriminative ability of confidence estimates. Theoretical analysis and extensive experiments across 15 real-world and synthetic datasets demonstrate that CSR consistently exhibits superior sensitivity and specificity across diverse calibration scenarios, validating the effectiveness and robustness of the proposed approach.

This study addresses the inconsistency of existing prediction evaluation metrics—such as ABC and Gini—with the principle of mean consistency, stemming from their reliance on predicted values for weighting, which can lead to erroneous model selection. Building upon Bregman divergences, the authors develop a mean-consistent loss framework, rederive the Murphy decomposition to disentangle prediction error into calibration and discrimination components, and establish a theoretical link between these components and Lorenz-curve-based metrics. They propose a new metric, ABC², to enhance sensitivity to mean calibration, and demonstrate that ABC, ABC², and Gini all violate mean consistency due to prediction-dependent weighting. Furthermore, they prove the equivalence between the number of crossings in Lorenz and Murphy curves and, under a single-crossing condition, provide a weak dominance criterion for predictive superiority, offering both theoretical grounding and practical guidelines for reliable model evaluation.

This work addresses the instability of validation rankings in class-imbalanced settings caused by threshold dependence in conventional Matthews correlation coefficient (MCC)–based model selection. The authors propose SoftMCC, a threshold-free framework that constructs an MCC-based validation metric using probabilistic outputs. Under perfect calibration, SoftMCC unifies MCC with the Brier skill score and introduces a covariance-normalized measure of association between probabilities and labels, which retains Pearson’s boundedness and reduces to hard-prediction MCC in the deterministic limit. Integrating probabilistic confusion counts, temperature scaling, and tie-aware shared pool selection, SoftMCC achieves the best average rank (2.31) and highest Kendall’s W (0.659) across 18 experimental settings, significantly outperforming AUPRC and MCC@0.5 in ranking stability. However, it remains sensitive to miscalibration and yields only modest gains in test-set MCC.