Online Conformal Model Selection for Nonstationary Time Series

2025-06-05

AI Summary

Traditional model selection methods (e.g., AIC, cross-validation) rely on stationarity assumptions and fail under nonstationary time series. To address this, we propose the Model Prediction Set (MPS) framework—the first online model selection method that integrates conformal inference with model confidence sets. MPS dynamically constructs, in real time, a statistically valid confidence set containing the optimal model for the next time step, even under unknown distributional shifts (gradual or abrupt), while providing long-term coverage guarantees. Crucially, MPS imposes no stationarity assumptions, accommodates arbitrary model classes and evaluation metrics, and supports efficient online updates. Experiments on synthetic and real-world datasets demonstrate that MPS consistently identifies the optimal model, yields compact yet high-quality candidate sets, accurately uncovers model evolution patterns, and significantly outperforms offline benchmark methods.

Technology Category

• Machine Learning: Online Learning & Bandits
• Reasoning under Uncertainty: Stochastic Optimization
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

This paper introduces the MPS (Model Prediction Set), a novel framework for online model selection for nonstationary time series. Classical model selection methods, such as information criteria and cross-validation, rely heavily on the stationarity assumption and often fail in dynamic environments which undergo gradual or abrupt changes over time. Yet real-world data are rarely stationary, and model selection under nonstationarity remains a largely open problem. To tackle this challenge, we combine conformal inference with model confidence sets to develop a procedure that adaptively selects models best suited to the evolving dynamics at any given time. Concretely, the MPS updates in real time a confidence set of candidate models that covers the best model for the next time period with a specified long-run probability, while adapting to nonstationarity of unknown forms. Through simulations and real-world data analysis, we demonstrate that MPS reliably and efficiently identifies optimal models under nonstationarity, an essential capability lacking in offline methods. Moreover, MPS frequently produces high-quality sets with small cardinality, whose evolution offers deeper insights into changing dynamics. As a generic framework, MPS accommodates any data-generating process, data structure, model class, training method, and evaluation metric, making it broadly applicable across diverse problem settings.

Problem

• Online model selection for nonstationary time series
• Adaptive model selection under unknown dynamic changes
• Real-time confidence sets for optimal evolving models

Innovation

• Combines conformal inference with model confidence sets
• Adaptively selects models for nonstationary time series
• Updates model sets in real time with specified probability