In-Context Learning Under Regime Change

2026-04-18

AI Summary

This study addresses the challenge of real-time adaptation in non-stationary time series under regime changes—abrupt shifts in data-generating mechanisms—without requiring model retraining. The authors formalize in-context change-point detection as a novel task and introduce an adaptive Transformer architecture capable of adjusting its behavior based on varying levels of prior knowledge about change points, ranging from completely unknown to precisely specified. Theoretical analysis reveals a fundamental trade-off between model complexity and the informativeness of change-point priors. By integrating in-context learning, change-point detection, and linear dynamical system modeling, the proposed method fine-tunes pretrained models using contextual information. It outperforms existing baselines on synthetic benchmarks and demonstrates substantial performance gains in real-world applications, including infectious disease forecasting and predicting financial market volatility around FOMC announcements.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Cognitive Modeling & Cognitive Systems: Adaptive Behavior
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Non-stationary sequences arise naturally in control, forecasting, and decision-making. The data-generating process shifts at unknown times, and models must detect the change, discard or downweight obsolete evidence, and adapt to new dynamics on the fly. Transformer-based foundation models increasingly rely on in-context learning for time series forecasting, tabular prediction, and continuous control. As these models are deployed in non-stationary environments, understanding their ability to detect and adapt to regime shifts is important. We formalize this as an in-context change-point detection problem and formally establish the existence of transformer models that solve this problem. Our construction demonstrates that model complexity, in layers and parameters, depends on the level of information available about the change-point location, from no knowledge to knowing exact timing. We validate our results with experiments on synthetic linear regression and linear dynamical systems, where trained transformers match the performance of optimal baselines across information levels. We also show that encoding and incorporating changepoint knowledge indeed improves the real-world performance of a pretrained foundation models on infectious disease forecasting and on financial volatility forecasting around Federal Open Market Committee (FOMC) announcements without retraining, demonstrating practical applicability to real-world regime changes.

Problem

• in-context learning
• regime change
• non-stationary
• change-point detection
• transformer

Innovation

• in-context learning
• regime change
• change-point detection
• non-stationary time series
• Transformer adaptation