The Exchangeability Assumption for Permutation Tests of Multiple Regression Models: Implications for Statistics and Data Science

2024-06-11

AI Summary

This paper addresses the often-overlooked exchangeability assumption underlying permutation tests in multiple linear regression—a condition critical for valid statistical inference. We rigorously clarify the logical relationship between exchangeability and the null hypothesis, and systematically evaluate the robustness of common permutation schemes—response permutation, residual permutation, and design-matrix permutation—under both satisfied and violated exchangeability conditions, via theoretical analysis and simulation studies. We propose a novel, pedagogically integrated framework that unifies conceptual understanding with formal theory, and extend the analysis for the first time to hierarchical and clustered regression models, enhancing methodological generality. Results show that while standard regression settings yield consistent conclusions across permutation schemes, inference deteriorates markedly when exchangeability fails. Our framework significantly improves students’ conceptual grasp of resampling-based inference, offering a new paradigm for statistics education and applied practice.

Technology Category

• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Machine Learning: Calibration & Uncertainty Quantification
• Knowledge Representation and Reasoning: Argumentation

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

Permutation tests are a powerful and flexible approach to inference via resampling. As computational methods become more ubiquitous in the statistics curriculum, use of permutation tests has become more tractable. At the heart of the permutation approach is the exchangeability assumption, which determines the appropriate null sampling distribution. We explore the exchangeability assumption in the context of permutation tests for multiple linear regression models. Various permutation schemes for the multiple linear regression setting have been previously proposed and assessed in the literature. As has been demonstrated previously, in most settings, the choice of how to permute a multiple linear regression model does not materially change inferential conclusions. Regardless, we believe that (1) understanding exchangeability in the multiple linear regression setting and also (2) how it relates to the null hypothesis of interest is valuable. We also briefly explore model settings beyond multiple linear regression (e.g., settings where clustering or hierarchical relationships exist) as a motivation for the benefit and flexibility of permutation tests. We close with pedagogical recommendations for instructors who want to bring multiple linear regression permutation inference into their classroom as a way to deepen student understanding of resampling-based inference.

Problem

• Examining exchangeability assumption in permutation tests for regression
• Assessing impact of permutation schemes on Type I errors
• Providing pedagogical guidance for teaching regression permutation inference

Innovation

• Permutation tests for multiple regression models
• Assessing exchangeability assumption in permutations
• Pedagogical recommendations for permutation inference