CODA: Coordination via On-Policy Diffusion for Multi-Agent Offline Reinforcement Learning

2026-04-25

AI Summary

Offline multi-agent reinforcement learning often converges to suboptimal equilibria due to the static nature of offline datasets, which impedes effective co-adaptation of policies. To address this challenge, this work proposes the first policy-conditioned online diffusion-based trajectory generator that dynamically synthesizes multi-agent experiences conditioned on the current joint policy, thereby enabling data augmentation synchronized with policy evolution. The approach is algorithm-agnostic and seamlessly integrates with both model-based and model-free frameworks, facilitating coordinated adaptation among agents. Empirical results demonstrate that the method successfully overcomes common coordination failures in continuous polynomial games and achieves significant performance gains on complex control tasks in the Multi-Agent MuJoCo (MaMuJoCo) benchmark.

Technology Category

• Multiagent Systems: Adversarial Agents
• Machine Learning: Online Learning & Bandits
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Offline multi-agent reinforcement learning (MARL) enables policy learning from fixed datasets, but is prone to coordination failure: agents trained on static, off-policy data converge to suboptimal joint behaviours because they cannot co-adapt as their policies change. We introduce CODA (Coordination via On-Policy Diffusion for Multi-Agent Reinforcement Learning), a diffusion-based multi-agent trajectory generator for data augmentation that samples conditioned on the current joint policy, producing synthetic experience which reflects the evolving behaviours of the agents, thereby providing a mechanism for co-adaptation. We find that previous diffusion-based augmentation approaches are insufficient for fostering multi-agent coordination because they produce static augmented datasets that do not evolve as the current joint policy changes during training; CODA resolves this by more closely simulating on-policy learning and is a meaningful step toward coordinated behaviours in the offline setting. CODA is algorithm-agnostic and can be layered onto both model-free and model-based offline reinforcement learning pipelines as an augmentation module. Empirically, CODA not only resolves canonical coordination pathologies in continuous polynomial games but also delivers strong results on the more complex MaMuJoCo continuous-control benchmarks.

Problem

• offline multi-agent reinforcement learning
• coordination failure
• co-adaptation
• static datasets
• suboptimal joint behaviours

Innovation

• on-policy diffusion
• multi-agent coordination
• offline reinforcement learning
• trajectory generation
• data augmentation