More Permutations Do Not Always Increase Power: Non-monotonicity in Monte Carlo Permutation Tests

2026-05-05

AI Summary

This study challenges the common belief that increasing the number of permutations in Monte Carlo permutation tests invariably enhances statistical power. By integrating discrete distribution theory, probability theory, and Monte Carlo simulations, the authors rigorously demonstrate—for the first time—that under standard settings, statistical power exhibits a non-monotonic, sawtooth-like behavior due to the discreteness of the underlying distribution, leading to infinitely many decreases as the number of permutations grows. These findings reveal a complex relationship between computational budget and test power, offering crucial theoretical corrections for the practical implementation of permutation tests.

Abstract

Monte Carlo permutation tests are a cornerstone of valid, model-free statistical inference. A widely held practical intuition is that increasing the number of sampled permutations improves test performance, in particular that statistical power tends to increase with the Monte Carlo budget. In this paper, we show that these intuitions are false in general. Leveraging the saw-toothed structure of power arising from distributional discreteness, we provide a simple structural explanation for why power can decrease as the number of sampled permutations increases, and we prove that such decreases occur infinitely often as the Monte Carlo budget grows.

Problem

• Monte Carlo permutation tests
• statistical power
• non-monotonicity
• discreteness
• permutation sampling

Innovation

• Monte Carlo permutation tests
• statistical power
• non-monotonicity
• distributional discreteness
• saw-toothed structure