The Cell Must Go On: Agar.io for Continual Reinforcement Learning

2025-05-23

AI Summary

Existing continual reinforcement learning (CRL) benchmarks are largely confined to episodic, low-dimensional, or fully observable environments, limiting the study of progressive behavioral evolution and dynamic adaptation. To address this, we propose AgarCL—the first CRL benchmark built upon Agar.io, a non-episodic, high-dimensional, partially observable open-world game. Its core innovations include: (i) the first adaptation of a realistic open-world game into a CRL benchmark; (ii) support for online environmental evolution, continuous action spaces, and multi-granularity task decoupling analysis; and (iii) integration of mainstream algorithms including DQN, PPO, and SAC. Our experiments deliver the first systematic evaluation of CRL in a high-complexity, non-episodic setting, clearly delineating performance boundaries across algorithms with respect to environmental dynamics, partial observability, and continual adaptability.

Technology Category

• Multiagent Systems: Adversarial Agents
• Game Theory and Economic Paradigms: Adversarial Learning
• Machine Learning: Life-Long and Continual Learning

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Economics, Online Markets and Human Computation: Fairness, privacy, and diversity in economic environments
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Continual reinforcement learning (RL) concerns agents that are expected to learn continually, rather than converge to a policy that is then fixed for evaluation. Such an approach is well suited to environments the agent perceives as changing, which renders any static policy ineffective over time. The few simulators explicitly designed for empirical research in continual RL are often limited in scope or complexity, and it is now common for researchers to modify episodic RL environments by artificially incorporating abrupt task changes during interaction. In this paper, we introduce AgarCL, a research platform for continual RL that allows for a progression of increasingly sophisticated behaviour. AgarCL is based on the game Agar.io, a non-episodic, high-dimensional problem featuring stochastic, ever-evolving dynamics, continuous actions, and partial observability. Additionally, we provide benchmark results reporting the performance of DQN, PPO, and SAC in both the primary, challenging continual RL problem, and across a suite of smaller tasks within AgarCL, each of which isolates aspects of the full environment and allow us to characterize the challenges posed by different aspects of the game.

Problem

• Developing continual RL agents for dynamic environments
• Addressing limitations in current continual RL simulators
• Benchmarking RL algorithms in non-episodic, complex settings

Innovation

• AgarCL platform for continual RL
• Based on Agar.io game dynamics
• Benchmarks DQN, PPO, SAC performance