De-meaning Simulation Studies

2026-06-18

AI Summary

Traditional simulation studies often rely on mean and standard deviation to assess the quality of asymptotic approximations; however, the existence and convergence of moments are not guaranteed by distributional convergence alone, and such approaches inadequately characterize near-normal approximations contaminated by outliers. This work proposes replacing conventional moment-based summaries with quantile-based summary statistics, specifically employing the median, median absolute deviation, and empirical confidence interval coverage as robust and universally applicable evaluation metrics. By shifting away from the mean-centered paradigm, this approach overcomes both theoretical and practical limitations inherent in moment-based methods and provides a more interpretable and reliable criterion for evaluating simulation results.

Abstract

In simulation studies evaluating asymptotic approximations it is common practice to report averages and standard deviations over repeated simulations. We argue that quantile-based summaries are more appropriate from both a theoretical and practical point of view. Theoretically, convergence of moments -- or even existence of moments -- is not guaranteed by convergence in distribution, so sample moments are not ideal for assessing the accuracy of a distributional approximation. In practice, means and variances are not good summaries of approximately-Normal distributions that may have occasional outliers. We suggest the median and median absolute deviation, and empirical confidence interval coverage, as better general summaries, and argue that moments should be reserved for simulation settings where they are of substantive interest.

Problem

• simulation studies
• asymptotic approximations
• quantile-based summaries
• distributional convergence
• moments

Innovation

• quantile-based summaries
• simulation studies
• asymptotic approximations
• median absolute deviation
• empirical coverage