LOO-PIT predictive model checking

2026-03-03

AI Summary

This work addresses the reduced power of conventional uniformity tests for leave-one-out probability integral transform (LOO-PIT) values in Bayesian model calibration, which arises from their inherent dependence structure. To overcome this limitation, the authors propose a calibration diagnostic framework that explicitly accounts for the dependence among LOO-PIT values under finite-sample settings. By elucidating the relationship between LOO-PIT dependence and model complexity, they develop three dependence-aware uniformity tests applicable to both continuous and discrete data, complemented by an automated visualization tool to detect localized calibration discrepancies. Experiments on simulated and real-world datasets demonstrate that the proposed methods substantially outperform traditional independence-based tests, significantly enhancing the ability to identify model misspecification.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Relational Probabilistic Models
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

We consider predictive checking for Bayesian model assessment using leave-one-out probability integral transform (LOO-PIT). LOO-PIT values are conditional cumulative predictive probabilities given LOO predictive distributions and corresponding left out observations. For a well-calibrated model, LOO-PIT values should be near uniformly distributed, but in the finite sample case they are not independent, due to LOO predictive distributions being determined by nearly the same data (all but one observation). We prove that this dependency is non-negligible in the finite case and depends on model complexity. We propose three testing procedures that can be used for continuous and discrete dependent uniform values. We also propose an automated graphical method for visualizing local departures from the null. Extensive numerical experiments on simulated and real datasets demonstrate that the proposed tests achieve competitive performance overall and have much higher power than standard uniformity tests based on the independence assumption that inevitably lead to lower than expected rejection rate.

Problem

• LOO-PIT
• Bayesian model assessment
• predictive checking
• uniformity test
• dependence

Innovation

• LOO-PIT
• Bayesian model assessment
• dependent uniformity test
• leave-one-out predictive checking
• model calibration