Calibration without labels in multiple testing

2026-06-17

AI Summary

This study addresses the long-standing challenge of calibrating local false discovery rates (lfdr) in multiple hypothesis testing when true labels are unavailable. The authors introduce, for the first time, a pseudo-labeling mechanism based on spacings between ordered p-values, reframing lfdr calibration as an unsupervised regression problem and thereby enabling the application of classical probability calibration tools. By integrating an empirical Bayes framework with posterior calibration techniques, the method reveals that the widely used q-value approach suffers from substantial miscalibration. Extensive empirical analyses in psychology and neuroscience demonstrate that the proposed approach significantly enhances the reliability and interpretability of lfdr estimates, underscoring its necessity and superiority over existing practices.

Abstract

Large-scale hypothesis testing supports probability claims about individual hypotheses, as in empirical Bayes methods for estimating local false discovery rates. We study how such claims can be interpreted as approximately calibrated forecasts of the null hypothesis, yielding interpretable error probabilities even under model misspecification. Our approach draws conceptual inspiration from probabilistic forecasting but addresses a different challenge: unlike forecasting, where labels are eventually observed, in multiple testing the ground truth is never revealed, so calibration must be assessed stochastically and established indirectly. We address this challenge by constructing a set of pseudo-labels, derived from the spacings of ordered $p$-values, which have the local false discovery rate as their regression target. Our construction unlocks existing tools for assessing and performing post-hoc calibration in multiple testing. Notably, we find on a large-scale empirical survey of published psychology and neuroscience literature that the $q$-value, a popular error measure based on the false discovery rate, can be severely miscalibrated.

Problem

• multiple testing
• calibration
• false discovery rate
• local false discovery rate
• pseudo-labels

Innovation

• calibration without labels
• multiple testing
• local false discovery rate
• pseudo-labels
• q-value miscalibration