R2D-RL: A RoboCup 2D Soccer Environment for Multi-Agent Reinforcement Learning

2026-06-17

AI Summary

This work addresses the challenge of integrating the RoboCup 2D soccer simulation platform into modern Python-based multi-agent reinforcement learning (MARL) workflows. To bridge this gap, we propose an efficient integration framework that seamlessly connects the HELIOS client to mainstream MARL libraries through shared-memory communication and cycle-level synchronization. The resulting environment supports full-field and offensive-half scenarios, configurable opponents, hybrid action spaces, action masking, and reward shaping based on Expected Possession Value (EPV), while enabling highly parallelized training. As the first native integration of RoboCup 2D with contemporary MARL frameworks, our approach significantly enhances algorithm development efficiency and experimental reproducibility, establishing a standardized and scalable benchmark platform for MARL research.

Abstract

Robot soccer is a challenging testbed for multi-agent reinforcement learning because it combines partial observability, cooperative and adversarial interaction, sparse rewards, and long-horizon tactical behavior. RoboCup 2D Soccer Simulation (RCSS2D) provides a mature robot-soccer platform, but its competition-oriented server-client architecture is difficult to use directly with modern Python-based MARL workflows. We introduce R2D-RL, a reinforcement learning environment that connects RCSS2D and HELIOS-based player clients to a Python MARL interface through shared-memory communication and cycle-level synchronization. R2D-RL supports full-field and scenario-based training with configurable opponents, Base discrete and Hybrid parameterized action spaces, action masks, expected possession value (EPV)-based reward shaping, and parallel execution. We provide front-goal scenarios and an 11-vs-11 full-field benchmark, together with baseline results.

Problem

• RoboCup 2D Soccer
• multi-agent reinforcement learning
• Python MARL interface
• server-client architecture
• simulation environment

Innovation

• multi-agent reinforcement learning
• RoboCup 2D Soccer
• shared-memory communication
• parameterized action spaces
• reward shaping