Combining Reinforcement Learning and Behavior Trees for NPCs in Video Games with AMD Schola

2025-10-15

AI Summary

Reinforcement learning (RL) faces challenges in commercial game NPC deployment, including poor interpretability, difficulty supporting long-horizon strategic planning, and limited multi-task coordination. Method: This paper proposes a tightly integrated RL–Behavior Tree (BT) framework. Leveraging the AMD Schola plugin, we implement a closed-loop training system in Unreal Engine, combining Proximal Policy Optimization (PPO), curriculum learning, and a hierarchical reward structure to enable node-level BT policy optimization and global behavioral coordination. Contribution/Results: To our knowledge, this is the first work to demonstrate end-to-end RL-BT co-training in a complex 3D open-world environment inspired by *The Last of Us*. Experiments show that trained NPCs autonomously execute exploration, close- and long-range combat, and squad-level cooperation; dynamically switch skills; and plan over extended time horizons. The approach significantly improves behavioral adaptability, interpretability, and perceptual realism, establishing a novel paradigm for industrial-scale game AI deployment.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Reinforcement Learning
• Multiagent Systems: Multiagent Learning

Application Category

• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

While the rapid advancements in the reinforcement learning (RL) research community have been remarkable, the adoption in commercial video games remains slow. In this paper, we outline common challenges the Game AI community faces when using RL-driven NPCs in practice, and highlight the intersection of RL with traditional behavior trees (BTs) as a crucial juncture to be explored further. Although the BT+RL intersection has been suggested in several research papers, its adoption is rare. We demonstrate the viability of this approach using AMD Schola -- a plugin for training RL agents in Unreal Engine -- by creating multi-task NPCs in a complex 3D environment inspired by the commercial video game ``The Last of Us". We provide detailed methodologies for jointly training RL models with BTs while showcasing various skills.

Problem

• Combining reinforcement learning with behavior trees for NPCs
• Addressing slow adoption of RL in commercial video games
• Training multi-task NPCs in complex 3D game environments

Innovation

• Combining reinforcement learning with behavior trees
• Using AMD Schola plugin for training agents
• Creating multi-task NPCs in complex environments