Adaptive Agent Design

2026-09-11

AI Summary

研究了非马尔可夫环境中自适应代理设计问题，通过软Q学习算法优化状态转移核和策略，并分析了部分可观测马尔可夫决策问题中的收敛性。

Abstract

We consider an agent acting against a general non-Markovian environment. The agent maintains its agent states, but is free to choose a transition kernel across those states and optimize its state-feedback control policies. We study the bi-level agent design problem that optimizes the transition kernel and the policy it induces, given said kernel with offline data of observations and actions obtained via a behavioral policy. For general environments, we show that a soft $Q$-learning algorithm converges almost surely to the fixed point of a soft Bellman equation defined by the stationary averages that the behavioral policy and the chosen kernel induce, and we delineate what separates the resulting policy from an optimal one. In partially observed Markov decision problems, we analyze convergence properties of parametrized transition kernel design via zero-th order and Bayesian optimization techniques.

Problem

• Adaptive Agent
• Non-Markovian Environment
• Transition Kernel
• State-Feedback Control Policy
• Offline Data

Innovation

• Adaptive Agent Design
• Soft Q-learning
• Non-Markovian Environment
• Transition Kernel Optimization
• Bayesian Optimization