🤖 AI Summary
This work addresses the adaptive optimal control of linear diffusion processes—governed by linear stochastic differential equations—with unknown drift matrices, a setting prevalent in continuous-decision domains such as medical intervention and flight control where system dynamics are uncertain. We propose the first method embedding Thompson sampling into a continuous-time stochastic optimal control framework. Theoretically, it achieves an $ ilde{O}(sqrt{T})$ cumulative regret bound and guarantees short-term system stability. A key conceptual innovation is the introduction of the “optimality manifold,” a geometric construct enabling local analysis of parameter sensitivity and stability boundaries. Extensive simulations demonstrate superior performance over baseline algorithms in aircraft attitude regulation and artificial pancreas blood glucose control tasks, particularly improving worst-case regret.
📝 Abstract
Diffusion processes that evolve according to linear stochastic differential equations are an important family of continuous-time dynamic decision-making models. Optimal policies are well-studied for them, under full certainty about the drift matrices. However, little is known about data-driven control of diffusion processes with uncertain drift matrices as conventional discrete-time analysis techniques are not applicable. In addition, while the task can be viewed as a reinforcement learning problem involving exploration and exploitation trade-off, ensuring system stability is a fundamental component of designing optimal policies. We establish that the popular Thompson sampling algorithm learns optimal actions fast, incurring only a square-root of time regret, and also stabilizes the system in a short time period. To the best of our knowledge, this is the first such result for Thompson sampling in a diffusion process control problem. We validate our theoretical results through empirical simulations with real parameter matrices from two settings of airplane and blood glucose control. Moreover, we observe that Thompson sampling significantly improves (worst-case) regret, compared to the state-of-the-art algorithms, suggesting Thompson sampling explores in a more guarded fashion. Our theoretical analysis involves characterization of a certain optimality manifold that ties the local geometry of the drift parameters to the optimal control of the diffusion process. We expect this technique to be of broader interest.
