Conformalized Multiple Testing under Unknown Null Distribution with Symmetric Errors

2025-09-04

AI Summary

In large-scale multiple testing, deviation between the theoretical null distribution and the true null distribution often leads to loss of FDR control. To address this, we propose Self-Calibrated Empirical Bayes (SC-EB), a novel method that relaxes the conventional Gaussian assumption—requiring only symmetry of the error distribution. SC-EB constructs Self-calibrated Empirical Null Samples (SENS) and integrates conformal inference to guarantee finite-sample FDR control while preserving the statistical power of empirical Bayes. It unifies single- and two-sample testing frameworks and robustly mitigates both null distribution misspecification and estimation bias. Extensive simulations and real-data analyses demonstrate that SC-EB significantly outperforms existing empirical Bayes and model-free FDR methods in both FDR robustness and detection power.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Search and Optimization: Sampling/Simulation-based Search
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

This article addresses a fundamental concern, first raised by Efron (2004), regarding the selection of null distributions in large-scale multiple testing. In modern data-intensive applications involving thousands or even millions of hypotheses, the theoretical null distribution of the test statistics often deviates from the true underlying null distribution, severely compromising the false discovery rate (FDR) analysis. We propose a conformalized empirical Bayes method using self-calibrated empirical null samples (SENS) for both one-sample and two-sample multiple testing problems. The new framework not only sidesteps the use of potentially erroneous theoretical null distributions, which is common in conventional practice, but also mitigates the impact of estimation errors in the unknown null distribution on the validity of FDR control, a challenge frequently encountered in the empirical Bayes FDR literature. In contrast to the empirical Bayes approaches (cf. Efron, 2004; Jin and Cai, 2007; Sun and Cai, 2007) that rely on Gaussian assumptions for the null models, SENS imposes only a weak condition on the symmetry of the error distribution, and leverages conformal tools to achieve FDR control in finite samples. Moreover, SENS incorporates structural insights from empirical Bayes into inference, exhibiting higher power compared to frequentist model-free methods. We conduct an in-depth analysis to establish a novel optimality theory for SENS under Efron's two-group model and demonstrate its superiority over existing empirical Bayes FDR methods and recent model-free FDR methods through numerical experiments on both simulated and real data.

Problem

• Addresses incorrect null distribution selection in large-scale multiple testing
• Proposes conformalized method to control false discovery rate accurately
• Mitigates estimation errors in unknown null distribution for valid inference

Innovation

• Conformalized empirical Bayes method using self-calibrated null samples
• Leverages symmetry of error distribution with conformal tools
• Achieves finite-sample FDR control without Gaussian assumptions