Near-Optimal Reinforcement Learning with Multi-Step Transition Lookahead

2026-09-10

AI Summary

研究了带有多个步骤转换预览的强化学习问题，证明了其在固定折扣因子下的NP难性，并提出了一种近似方案以实现高效近优规划。

Abstract

We study reinforcement learning (RL) with transition look-ahead, where the agent may observe which states would be visited upon playing any sequence of $\ell$ actions before deciding its course of action. Although look-ahead can substantially improve achievable performance, it is known that optimal planning with multi-step transition look-ahead is NP-hard, but this hardness was established using discount factors arbitrarily close to one. It was therefore unknown whether the problem remains hard for any discount factor, and whether near-optimal planning can nevertheless be performed efficiently. We resolve both questions. First, we show that for every fixed rational discount factor ($γ\in(0,1)$), exact planning remains NP-hard. Second, we introduce a randomized polynomial-time approximation scheme for every fixed look-ahead depth. We then extend our approach to unknown transitions and stochastic rewards using optimism and variance-adaptive confidence bounds. The resulting algorithm achieves cumulative regret whose leading term matches classical tabular discounted RL up to logarithmic factors. Thus, although exact planning with transition look-ahead is NP-hard, efficient near-optimal planning and learning remain possible.

Problem

• reinforcement learning
• transition lookahead
• NP-hard
• discount factor
• near-optimal planning

Innovation

• reinforcement learning
• multi-step transition lookahead
• randomized polynomial-time approximation scheme
• optimism and variance-adaptive confidence bounds