Superhuman AI for Generals.io Using Self-Play Reinforcement Learning

2026-06-22

AI Summary

This work achieves superhuman decision-making in Generals.io, a real-time strategy game characterized by strong imperfect information, effectively balancing long-term planning with immediate tactical execution. We propose an end-to-end vision Transformer policy network trained exclusively via self-play reinforcement learning using sparse win/loss rewards. Key innovations include a high-performance native simulator implemented in JAX—attaining tens of millions of frames per second throughput on a single GPU—combined with an advantage-based experience filtering mechanism and exponential moving average of policy parameters, which together overcome critical data efficiency bottlenecks. Trained for only four days, the model secured first place on the public leaderboard among over 5,000 players and decisively defeated the top two human competitors with a record of 199 wins to 70 losses, surpassing the performance gap between ranks 2 and 25 by a significant margin.

Abstract

We present a superhuman AI agent for Generals.io, a real-time strategy game that requires both long-horizon planning and short-term tactics under strong imperfect information. Trained for four days on 4x NVIDIA H200 GPUs, our agent reaches #1 on the public 1v1 leaderboard of over 5,000 human players, leading the second-ranked player by the same margin that separates second place from 25th, and beats the two top-ranked humans head-to-head with a combined 199-70 record across 269 ladder matches. A key enabler is a JAX-native simulator that reaches tens of millions of frames per second on a single GPU, roughly a 10,000x speedup over the prior simulator. On top of this, we train a vision transformer policy end-to-end by self-play with a policy-gradient loop and sparse win/loss reward, using top-advantage sample filtering and an exponential moving average of the policy parameters. Taken together, our findings highlight what matters, and what does not, once a fast simulator removes the data bottleneck.

Problem

• real-time strategy
• imperfect information
• long-horizon planning
• short-term tactics
• superhuman AI

Innovation

• self-play reinforcement learning
• JAX-native simulator
• vision transformer
• sparse reward
• top-advantage filtering