Effective sample size estimation based on concordance between p-value and posterior probability of the null hypothesis

2025-07-22

AI Summary

Prior effective sample size (ESS) estimation has long suffered from three key limitations: neglecting prior–likelihood compatibility, reliance on arbitrary benchmark priors, and inability to detect deleterious priors. This paper introduces a novel ESS estimator grounded in consistency testing between p-values and posterior probabilities under the null hypothesis—marking the first integration of ESS into a formal hypothesis-testing framework. Our method (1) explicitly models prior–likelihood compatibility, permitting negative ESS values to quantify harmful priors; (2) eliminates dependence on benchmark priors; and (3) enables unified assessment of synergistic effects across multiple priors. Bridging frequentist and Bayesian paradigms, the approach is validated through theoretical derivation, extensive simulations, and real-world eQTL data analysis. It consistently outperforms existing estimators and empirically demonstrates tangible gains from informative priors in genetic locus discovery.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Search and Optimization: Sampling/Simulation-based Search
• Reasoning under Uncertainty: Probabilistic Inference

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success

Abstract

Estimating the effective sample size (ESS) of a prior distribution is an age-old yet pivotal challenge, with great implications for clinical trials and various biomedical applications. Although numerous endeavors have been dedicated to this pursuit, most of them neglect the likelihood context in which the prior is embedded, thereby considering all priors as "beneficial". In the limited studies of addressing harmful priors, specifying a baseline prior remains an indispensable step. In this paper, by means of the elegant bridge between the p-value and the posterior probability of the null hypothesis, we propose a new ESS estimation method based on p-value in the framework of hypothesis testing, expanding the scope of existing ESS estimation methods in three key aspects: (i) We address the specific likelihood context of the prior, enabling the possibility of negative ESS values in case of prior-likelihood disconcordance; (ii) By leveraging the well-established bridge between the frequentist and Bayesian configurations under noninformative priors, there is no need to specify a baseline prior which incurs another criticism of subjectivity; (iii) By incorporating ESS into the hypothesis testing framework, our $p$-value ESS estimation method transcends the conventional one-ESS-one-prior paradigm and accommodates one-ESS-multiple-priors paradigm, where the sole ESS may reflect the collaborative impact of multiple priors in diverse contexts. Through comprehensive simulation analyses, we demonstrate the superior performance of the p-value ESS estimation method in comparison with existing approaches. Furthermore, by applying this approach to an expression quantitative trait loci (eQTL) data analysis, we show the effectiveness of informative priors in uncovering gene eQTL loci.

Problem

• Estimating effective sample size (ESS) for prior distributions
• Addressing prior-likelihood discordance with negative ESS values
• Expanding ESS estimation to multiple priors in hypothesis testing

Innovation

• Estimates ESS using p-value and posterior probability
• Enables negative ESS for prior-likelihood discordance
• Eliminates need for subjective baseline prior specification