Curriculum Learning With Counterfactual Group Relative Policy Advantage For Multi-Agent Reinforcement Learning

2025-06-09

AI Summary

To address training instability and suboptimal policies in multi-agent reinforcement learning (MARL) caused by static opponent policies and non-stationary environments, this paper proposes a dynamic curriculum learning framework. It adaptively adjusts opponent difficulty based on agents’ real-time performance and—novelly—integrates counterfactual causal reasoning into curriculum design via a Counterfactual Group-Relative Policy Advantage (CGRPA) mechanism. CGRPA enables precise individual contribution attribution and stable credit assignment under non-stationarity by jointly leveraging curriculum learning, counterfactual advantage estimation, and intra-group relative policy evaluation. The method significantly improves training stability and convergence speed, outperforming state-of-the-art MARL algorithms on the SMAC benchmark. The implementation is publicly available.

Technology Category

• Multiagent Systems: Adversarial Agents
• Game Theory and Economic Paradigms: Adversarial Learning
• Machine Learning: Causal Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Multi-agent reinforcement learning (MARL) has achieved strong performance in cooperative adversarial tasks. However, most existing methods typically train agents against fixed opponent strategies and rely on such meta-static difficulty conditions, which limits their adaptability to changing environments and often leads to suboptimal policies. Inspired by the success of curriculum learning (CL) in supervised tasks, we propose a dynamic CL framework for MARL that employs an self-adaptive difficulty adjustment mechanism. This mechanism continuously modulates opponent strength based on real-time agent training performance, allowing agents to progressively learn from easier to more challenging scenarios. However, the dynamic nature of CL introduces instability due to nonstationary environments and sparse global rewards. To address this challenge, we develop a Counterfactual Group Relative Policy Advantage (CGRPA), which is tightly coupled with the curriculum by providing intrinsic credit signals that reflect each agent's impact under evolving task demands. CGRPA constructs a counterfactual advantage function that isolates individual contributions within group behavior, facilitating more reliable policy updates throughout the curriculum. CGRPA evaluates each agent's contribution through constructing counterfactual action advantage function, providing intrinsic rewards that enhance credit assignment and stabilize learning under non-stationary conditions. Extensive experiments demonstrate that our method improves both training stability and final performance, achieving competitive results against state-of-the-art methods. The code is available at https://github.com/NICE-HKU/CL2MARL-SMAC.

Problem

• Dynamic curriculum learning for adaptive multi-agent reinforcement training
• Counterfactual credit assignment in non-stationary cooperative environments
• Stabilizing policy updates under evolving adversarial difficulty levels

Innovation

• Dynamic curriculum learning for adaptive difficulty
• Counterfactual group relative policy advantage
• Intrinsic credit signals for stable learning