Look-ahead Search on Top of Policy Networks in Imperfect Information Games

2023-12-23

International Joint Conference on Artificial Intelligence

AI Summary

To address the weak generalization and low search efficiency of reinforcement learning policies in imperfect-information games, this paper proposes a lightweight forward-search method applicable at test time: it requires only a policy network and an additionally trained critic network, without incorporating search during training. The method enables depth-limited, decision-point-agnostic look-ahead search via value estimation under policy transformations. Crucially, it seamlessly integrates critic-assisted search into any sampling-based policy gradient algorithm. We further design state abstraction and value modeling mechanisms specifically tailored to imperfect information. Empirical evaluation on Leduc Hold’em, multiple Goofspiel variants, and Battleships demonstrates substantial improvements in policy performance, validating the method’s effectiveness and scalability in large-scale imperfect-information games.

Abstract

Search in test time is often used to improve the performance of reinforcement learning algorithms. Performing theoretically sound search in fully adversarial two-player games with imperfect information is notoriously difficult and requires a complicated training process. We present a method for adding test-time search to an arbitrary policy-gradient algorithm that learns from sampled trajectories. Besides the policy network, the algorithm trains an additional critic network, which estimates the expected values of players following various transformations of the policies given by the policy network. These values are then used for depth-limited search. We show how the values from this critic can create a value function for imperfect information games. Moreover, they can be used to compute the summary statistics necessary to start the search from an arbitrary decision point in the game. The presented algorithm is scalable to very large games since it does not require any search during train time. We evaluate the algorithm's performance when trained along Regularized Nash Dynamics, and we evaluate the benefit of using the search in the standard benchmark game of Leduc hold'em, multiple variants of imperfect information Goofspiel, and Battleships.

Problem

• Imperfect Information Games
• Reinforcement Learning
• Strategy Network

Innovation

• Imperfect Information Games
• Strategy Gradient Algorithms
• Commentator Network