Flow-Corrected Thompson Sampling for Non-Stationary Contextual Bandits

2026-06-22

AI Summary

This work addresses the bias induced by time-varying reward models in non-stationary linear contextual bandits, where historical data become misaligned with the current environment. To tackle this challenge, the authors propose Flow-corrected Thompson Sampling (fcTS), which explicitly models parameter drift to dynamically recalibrate past rewards to the current distribution. By integrating confidence-weighted adaptive reuse of historical experiences, fcTS unifies the handling of three common non-stationarity patterns: gradual parameter drift, periodic variations, and abrupt state switches. Leveraging Bayesian inference, online slope estimation, phase-aware experience reuse, change-point detection, and truncated incremental sufficient statistics, fcTS enables closed-form posterior updates under linear Gaussian assumptions. Empirical evaluations demonstrate that fcTS significantly outperforms conventional forgetting-based approaches across five control tasks and a semi-synthetic portfolio benchmark, with the most pronounced gains observed in environments exhibiting recurrent temporal structures.

Abstract

We study non-stationary linear contextual bandits where the reward model drifts over time, rendering classical contextual bandit algorithms brittle because historical data becomes systematically biased. We propose Flow-Corrected Thompson Sampling (fcTS), a Bayesian method that reuses experience by transporting past rewards to the present using an explicit drift model and incorporating each transported observation with a confidence weight that reflects transport reliability. This yields a unified template that specializes in (i) linear parameter drift via online slope estimation and reward correction, (ii) periodic variation via phase-aware reuse across cycles, and (iii) recurring regime switches via changepoint detection and regime-specific posterior memory. The resulting posterior updates remain closed-form under a linear Gaussian model and can be implemented efficiently with truncated, incrementally updated sufficient statistics. Across five controlled case studies and a semi-synthetic portfolio-selection benchmark with multiple overlapping non-stationarities, fcTS outperforms standard forgetting-based baselines (discounting, sliding windows, and periodic restarts), with the largest gains in settings exhibiting recurring temporal structure. These results demonstrate that when non-stationarity is structured, correcting and reweighting historical observations can be substantially more sample-efficient than uniformly discarding them.

Problem

• non-stationary
• contextual bandits
• reward drift
• historical bias
• temporal structure

Innovation

• non-stationary contextual bandits
• Thompson Sampling
• drift correction
• structured non-stationarity
• Bayesian posterior updating