Coachable agents for interactive gameplay

2026-07-01

AI Summary

This work addresses the challenge of enabling intelligent agents to simultaneously achieve primary task objectives and dynamically adapt their behavioral style in response to user specifications during complex tasks. The authors propose a reinforcement learning framework based on Universal Value Function Approximators (UVFAs), augmented with tailored training scenarios and data augmentation strategies, which allows agents to flexibly adjust their behavior styles at runtime. This approach is the first to be successfully deployed in diverse and demanding environments—including Horizon Forbidden West, Gran Turismo, and an open-source humanoid robot platform—demonstrating high-fidelity style responsiveness while maintaining stable task performance. The results significantly enhance the behavioral flexibility and controllability of learned agents without compromising core task execution.

Abstract

Reinforcement learning has proven to be a valuable tool in the creation of advanced AI and robotic systems, contributing to everything from game playing to robotics to foundation models. Through trial-and-error, these AI systems typically learn one, near-optimal behavior to solve their tasks. However, there are many use cases in which one would like to assert some level of control, preferably in real time, over how the task is solved. We refer to these modifications of a core task as styles. We combine universal value function approximators (UVFAs) with carefully selected training scenarios, learning algorithms, and data augmentation to create a framework for coaching agents that exhibit styles in complex domains. We demonstrate the framework's application in the AAA video games Horizon Forbidden West and Gran Turismo, and in an open-source humanoid test domain. Despite the different nature of the domains -- car racing, stylized game combat, and humanoid walking -- each agent shows strong coherence to the style requests while still satisfying the main task in its domain. Importantly, the techniques outlined in this paper allow an end user to choose the final behavior at run time, giving them flexible control over the final executed performance.

Problem

• coachable agents
• behavior styles
• real-time control
• reinforcement learning
• task execution

Innovation

• coachable agents
• universal value function approximators
• style-conditioned reinforcement learning
• real-time behavior control
• multi-domain adaptation