🤖 AI Summary
In asynchronous multi-agent reinforcement learning (MARL), credit assignment remains challenging due to agents’ independent, aperiodic action triggering—making it difficult to accurately attribute marginal contributions to joint rewards. To address this, we propose the Virtual Synchronization Proxy (VSP) mechanism and Multiplicative Value Decomposition (MVD) algorithm. VSP establishes, for the first time, a theoretically grounded, consistent mapping from physically asynchronous actions to a virtual synchronous modeling space. MVD employs multiplicative interaction modeling and counterfactual reasoning to ensure task fairness and algorithmic convergence. Integrated within a unified framework combining value decomposition, synchronization abstraction, and interpretability-aware design, our approach significantly outperforms state-of-the-art methods across multiple high-difficulty asynchronous MARL benchmarks. Moreover, it enhances both the interpretability and generalizability of emergent cooperative policies.
📝 Abstract
Credit assignment is a critical problem in multi-agent reinforcement learning (MARL), aiming to identify agents' marginal contributions for optimizing cooperative policies. Current credit assignment methods typically assume synchronous decision-making among agents. However, many real-world scenarios require agents to act asynchronously without waiting for others. This asynchrony introduces conditional dependencies between actions, which pose great challenges to current methods. To address this issue, we propose an asynchronous credit assignment framework, incorporating a Virtual Synchrony Proxy (VSP) mechanism and a Multiplicative Value Decomposition (MVD) algorithm. VSP enables physically asynchronous actions to be virtually synchronized during credit assignment. We theoretically prove that VSP preserves both task equilibrium and algorithm convergence. Furthermore, MVD leverages multiplicative interactions to effectively model dependencies among asynchronous actions, offering theoretical advantages in handling asynchronous tasks. Extensive experiments show that our framework consistently outperforms state-of-the-art MARL methods on challenging tasks while providing improved interpretability for asynchronous cooperation.