On admissibility in post-hoc hypothesis testing

2025-08-01

AI Summary

Conventional hypothesis testing mandates a pre-specified significance level α, precluding adaptive adjustment of false-positive costs after data observation or principled response to exceptionally strong evidence against the null. Method: We introduce “post-hoc hypothesis testing,” a novel framework that relaxes the fixed-α constraint by formalizing testing via Γ-acceptability—a decision-theoretic criterion parameterized by adversary sets. Contribution/Results: We prove that any Γ-acceptable post-hoc test must employ an e-value as its test statistic; further, we provide a complete classification of Γ-acceptable tests according to the structure of admissible adversary sets. Unifying e-value theory, statistical decision theory, and classical hypothesis testing, our framework characterizes, for the first time, the structural properties and feasibility boundaries of post-hoc α-selection under point null hypotheses—thereby enhancing flexibility, interpretability, and practical adaptability of statistical inference.

Technology Category

• Game Theory and Economic Paradigms: Adversarial Learning
• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Decision/Utility Theory

Application Category

• Security and Privacy: Large-scale security measurements
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

The validity of classical hypothesis testing requires the significance level $α$ be fixed before any statistical analysis takes place. This is a stringent requirement. For instance, it prohibits updating $α$ during (or after) an experiment due to changing concern about the cost of false positives, or to reflect unexpectedly strong evidence against the null. Perhaps most disturbingly, witnessing a p-value $pllα$ vs $pleq α$ has no (statistical) relevance for any downstream decision-making. Following recent work of Grünwald (2024), we develop a theory of post-hoc hypothesis testing, enabling $α$ to be chosen after seeing and analyzing the data. To study "good" post-hoc tests we introduce $Γ$-admissibility, where $Γ$ is a set of adversaries which map the data to a significance level. A test is $Γ$-admissible if, roughly speaking, there is no other test which performs at least as well and sometimes better across all adversaries in $Γ$. For point nulls and alternatives, we prove general properties of any $Γ$-admissible test for any $Γ$ and show that they must be based on e-values. We also classify the set of admissible tests for various specific $Γ$.

Problem

• Develops theory for post-hoc hypothesis testing flexibility
• Introduces Γ-admissibility to evaluate post-hoc tests
• Proves properties of Γ-admissible tests using e-values

Innovation

• Post-hoc hypothesis testing allows flexible alpha adjustment
• Introduces Γ-admissibility for evaluating test performance
• Admissible tests must be based on e-values