Design-Based Inference for the AUC with Complex Survey Data

2026-03-30

AI Summary

This study addresses the underestimation of AUC variance by conventional bootstrap methods under complex survey designs, which ignore the sampling structure and thereby yield distorted inference. The authors propose the first systematic design-based inferential framework for AUC estimation with complex survey data, innovatively incorporating replication weight methods—such as jackknife and balanced repeated replication—for accurate variance estimation. This approach enables valid confidence interval construction and hypothesis testing, and has been implemented in the R package svyROC. Extensive simulations demonstrate that the proposed method achieves coverage probabilities close to nominal levels and maintains appropriate statistical power across a range of complex sampling designs. Its practical utility is further corroborated through an analysis of NHANES data.

Abstract

Complex survey data are usually collected following complex sampling designs. Accounting for the sampling design is essential to obtain unbiased estimates and valid inferences when analyzing complex survey data. The area under the receiver operating characteristic curve (AUC) is routinely used to assess the discriminative ability of predictive models for binary outcomes. However, valid inference for the AUC under complex sampling designs remains challenging. Although bootstrap techniques are widely applied under simple random sampling for variance estimation in this framework, traditional implementations do not account for complex designs. In this work, we propose a design-based framework for AUC inference. In particular, replicate weights methods are used to construct confidence intervals and hypothesis tests. The performance of replicate weights methods and the traditional non-design-based bootstrap for this purpose has been analyzed through an extensive simulation study. Design-based methods achieve coverage probabilities close to nominal levels and appropriate rejection rates under the null hypothesis. In contrast, the traditional non-design-based bootstrap method tends to underestimate the variance, leading to undercoverage and inflated rejection rates. Differences between methods decrease as the number of selected clusters per stratum increases. An application to data from the National Health and Nutrition Examination Survey (NHANES) illustrates the practical relevance of the proposed framework. The methods have been incorporated into the svyROC R package.

Problem

• AUC
• complex survey data
• design-based inference
• variance estimation
• bootstrap

Innovation

• design-based inference
• AUC
• complex survey data
• replicate weights
• variance estimation