permApprox: a general framework for accurate permutation p-value approximation

2026-02-26

AI Summary

This study addresses the limitations of permutation testing caused by a finite number of permutations, which often yields empirical p-values of zero or a coarse distribution, thereby compromising statistical validity and undermining multiple testing correction. To overcome this, the authors propose a tail modeling approach based on the Generalized Pareto Distribution (GPD), introducing a support constraint in GPD fitting for the first time to guarantee non-zero and valid extrapolated p-values. The framework integrates robust maximum likelihood estimation, data-driven threshold selection, and a hybrid treatment of discrete–continuous p-values to deliver a complete and reliable approximation scheme. Evaluations on both simulated and real-world single-cell RNA-seq and microbiome datasets demonstrate that the method produces accurate, robust, smooth, and interpretable p-value distributions—even with a limited number of permutations.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Game Theory and Economic Paradigms: Imperfect Information
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Security and Privacy: Large-scale security measurements

Abstract

Permutation procedures are common practice in hypothesis testing when distributional assumptions about the test statistic are not met or unknown. With only few permutations, empirical p-values lie on a coarse grid and may even be zero when the observed test statistic exceeds all permuted values. Such zero p-values are statistically invalid and hinder multiple testing correction. Parametric tail modeling with the Generalized Pareto Distribution (GPD) has been proposed to address this issue, but existing implementations can again yield zero p-values when the estimated shape parameter is negative and the fitted distribution has a finite upper bound. We introduce a method for accurate and zero-free p-value approximation in permutation testing, embedded in the permApprox workflow and R package. Building on GPD tail modeling, the method enforces a support constraint during parameter estimation to ensure valid extrapolation beyond the observed statistic, thereby strictly avoiding zero p-values. The workflow further integrates robust parameter estimation, data-driven threshold selection, and principled handling of hybrid p-values that are discrete in the bulk and continuous in the extreme tail. Extensive simulations using two-sample t-tests and Wilcoxon rank-sum tests show that permApprox produces accurate, robust, and zero-free p-value approximations across a wide range of sample and effect sizes. Applications to single-cell RNA-seq and microbiome data demonstrate its practical utility: permApprox yields smooth and interpretable p-value distributions even with few permutations. By resolving the zero-p-value problem while preserving accuracy and computational efficiency, permApprox enables reliable permutation-based inference in high-dimensional and computationally intensive settings.

Problem

• permutation testing
• p-value approximation
• zero p-values
• multiple testing correction
• Generalized Pareto Distribution

Innovation

• permutation testing
• p-value approximation
• Generalized Pareto Distribution
• zero-free p-values
• support constraint