Sequential model confidence sets

2024-04-29

AI Summary

Traditional model confidence sets rely on the fixed-sample assumption, rendering them inadequate for continuous model selection and uncertainty quantification under dynamic data streams. To address this, we introduce the first sequential extension of model confidence sets, proposing a dynamic model screening framework grounded in e-processes, confidence sequences, and sequential hypothesis testing. Our method requires no prespecified stopping rule and delivers, at any time, a nonasymptotic, time-uniform confidence set that provably covers the true optimal model subset with guaranteed nominal coverage probability. Unlike static approaches, it substantially enhances statistical robustness and real-time adaptability in online model evaluation. The framework provides both theoretical guarantees and practical tools for trustworthy model selection in streaming data analysis.

Technology Category

• Reasoning under Uncertainty: Sequential Decision Making
• Machine Learning: Calibration & Uncertainty Quantification
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Web Mining and Content Analysis: Models for Web evolution

Abstract

In most prediction and estimation situations, scientists consider various statistical models for the same problem, and naturally want to select amongst the best. Hansen et al. (2011) provide a powerful solution to this problem by the so-called model confidence set, a subset of the original set of available models that contains the best models with a given level of confidence. Importantly, model confidence sets respect the underlying selection uncertainty by being flexible in size. However, they presuppose a fixed sample size which stands in contrast to the fact that model selection and forecast evaluation are inherently sequential tasks where we successively collect new data and where the decision to continue or conclude a study may depend on the previous outcomes. In this article, we extend model confidence sets sequentially over time by relying on sequential testing methods. Recently, e-processes and confidence sequences have been introduced as new, safe methods for assessing statistical evidence. Sequential model confidence sets allow to continuously monitor the models' performances and come with time-uniform, nonasymptotic coverage guarantees.

Problem

• Extending model confidence sets for sequential data analysis
• Addressing fixed sample size limitation in model selection
• Providing time-uniform coverage guarantees for model performance

Innovation

• Extends model confidence sets sequentially
• Uses sequential testing methods
• Incorporates e-processes and confidence sequences