Towards Instance-Wise Calibration: Local Amortized Diagnostics and Reshaping of Conditional Densities (LADaR)

2022-05-29

AI Summary

Existing conditional density estimation methods lack instance-level probability calibration, relying instead on population-averaged calibration that assumes global consistency—unsuitable for applications requiring sample-specific uncertainty quantification. Method: We propose LADaR, a framework integrating the Cal-PIT algorithm to learn local probability-probability (PP) mappings and perform adaptive distribution reshaping for individual inputs (x). LADaR constructs localized PP plots from calibration data, combining nonparametric density reweighting with conditional generative modeling to enable interpretable, instance-level diagnostic assessment. Contribution/Results: By abandoning the global calibration assumption, LADaR achieves superior instance-level calibration fidelity. Evaluated on the photometric redshift benchmark challenge—a key weak gravitational lensing cosmology task—it outperforms all 11 baseline methods in conditional density estimation error, fulfilling stringent requirements for high-precision, sample-level probabilistic calibration in modern cosmological inference.

Abstract

There is a growing interest in conditional density estimation and generative modeling of a target $y$ given complex inputs $mathbf{x}$. However, off-the-shelf methods often lack instance-wise calibration -- that is, for individual inputs $mathbf{x}$, the individual estimated probabilities can be very different from the true probabilities, even when the estimates are reasonable when averaged over the entire population. This paper introduces the LADaR (Local Amortized Diagnostics and Reshaping of Conditional Densities) framework and proposes an algorithm called $ exttt{Cal-PIT}$ that produces interpretable local calibration diagnostics and includes a mechanism to recalibrate the initial model. Our $ exttt{Cal-PIT}$ algorithm learns a single local probability-probability map from calibration data to assess and quantify where corrections are needed across the feature space. When necessary, it reshapes the initial distribution into an estimate with approximate instance-wise calibration. We illustrate the LADaR framework by applying $ exttt{Cal-PIT}$ to synthetic examples, including probabilistic forecasting with sequences of images as inputs, akin to predicting the wind speed of tropical cyclones from satellite imagery. Our main science application is conditional density estimation of galaxy distances given imaging data (so-called photometric redshift estimation). On a benchmark photometric redshift data challenge, $ exttt{Cal-PIT}$ achieves better conditional density estimation (as measured by the conditional density estimation loss) than all 11 other literature methods tested. This demonstrates its potential for meeting the stringent photometric redshift requirements for next generation weak gravitational lensing analyses.

Problem

• Probability Calibration
• Density Estimation
• Astronomical Data Analysis

Innovation

• LADaR
• Cal-PIT
• Conditional Density Estimation