When Are Trade-Off Functions Testable from Finite Samples?

2026-05-11

AI Summary

This work addresses the generally infeasible problem of nonparametrically inferring the power trade-off function for testing two unknown distributions from finite samples. To render the problem tractable, the authors introduce a structural assumption of “exact realizability,” stipulating that the rejection region belongs to a specific class of sets. Within this framework, they establish that the set class having finite VC dimension is both necessary and sufficient for finite-sample testability. Leveraging this insight, they construct the first simultaneous confidence band with non-asymptotic error control and without reliance on asymptotic approximations. Their approach integrates the Neyman–Pearson lemma, VC dimension theory, and set-class approximation techniques, yielding tests that strictly control Type I error and achieve uniform power under realizable alternatives. In monotone likelihood ratio models, the method attains nearly tight local separation rates and extends to one-dimensional log-concave distributions.

Abstract

We study finite-sample inference for the trade-off function of two unknown probability distributions, the function that traces the optimal type I/type II error frontier in binary testing. Given samples from distributions $P$ and $Q$, we consider the problem of testing whether their trade-off function lies above a benchmark curve $f_0$ or falls below a weaker benchmark $f_1$. Without structural restrictions, this problem is impossible uniformly over nonparametric classes. We identify a sharp condition under which it becomes possible. The key structural assumption is that the Neyman--Pearson rejection regions for $(P,Q)$ are attainable, up to null sets, by a prescribed class $S$ of measurable sets. Within this exact attainability framework, finite Vapnik--Chervonenkis dimension of $S$ is both sufficient and necessary for nontrivial finite-sample testing. We construct a test with nonasymptotic error guarantees: type I error control is valid without assuming attainability, while power holds uniformly over attainable alternatives satisfying an explicit separation condition. By inverting the test, we also obtain simultaneous confidence bands for the whole trade-off curve. Finally, we study the sharpness and robustness of the procedure. In the monotone likelihood-ratio model, we derive local separation rates and prove matching lower bounds up to logarithmic factors. We also allow approximate, rather than exact, attainability; this extension yields finite-sample guarantees for univariate log-concave distributions by approximating their rejection regions with unions of intervals.

Problem

• trade-off function
• finite-sample inference
• hypothesis testing
• Neyman-Pearson rejection regions
• nonparametric statistics

Innovation

• trade-off function
• finite-sample inference
• VC dimension
• Neyman–Pearson rejection regions
• simultaneous confidence bands