Revisiting Multi-Agent World Modeling from a Diffusion-Inspired Perspective

2025-05-27

AI Summary

Addressing core challenges in multi-agent reinforcement learning (MARL)—including intractable environment modeling, exponential growth of the joint action space, and high dynamic uncertainty—this paper proposes a state-evolution-driven sequential world model. Instead of modeling joint state-action transitions, our approach exclusively captures the temporal evolution of states and progressively resolves uncertainty via agent-wise temporal decoupling. Inspired by diffusion models, we innovatively reinterpret their reverse generative process as a sequential revelation mechanism for multi-agent actions, thereby establishing the first diffusion-based paradigm for multi-agent world modeling. Evaluated on benchmarks including MAMuJoCo and Bi-DexHands, our method significantly outperforms state-of-the-art approaches, achieving substantial improvements in both sample efficiency and final policy return.

Technology Category

• Multiagent Systems: Adversarial Agents
• Reasoning under Uncertainty: Sequential Decision Making
• Computer Vision: Diffusion Models for Vision

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

World models have recently attracted growing interest in Multi-Agent Reinforcement Learning (MARL) due to their ability to improve sample efficiency for policy learning. However, accurately modeling environments in MARL is challenging due to the exponentially large joint action space and highly uncertain dynamics inherent in multi-agent systems. To address this, we reduce modeling complexity by shifting from jointly modeling the entire state-action transition dynamics to focusing on the state space alone at each timestep through sequential agent modeling. Specifically, our approach enables the model to progressively resolve uncertainty while capturing the structured dependencies among agents, providing a more accurate representation of how agents influence the state. Interestingly, this sequential revelation of agents' actions in a multi-agent system aligns with the reverse process in diffusion models--a class of powerful generative models known for their expressiveness and training stability compared to autoregressive or latent variable models. Leveraging this insight, we develop a flexible and robust world model for MARL using diffusion models. Our method, Diffusion-Inspired Multi-Agent world model (DIMA), achieves state-of-the-art performance across multiple multi-agent control benchmarks, significantly outperforming prior world models in terms of final return and sample efficiency, including MAMuJoCo and Bi-DexHands. DIMA establishes a new paradigm for constructing multi-agent world models, advancing the frontier of MARL research.

Problem

• Reducing modeling complexity in multi-agent reinforcement learning
• Accurately capturing agent dependencies in state transitions
• Improving sample efficiency and performance in MARL

Innovation

• Sequential agent modeling reduces complexity
• Diffusion models enhance world model accuracy
• DIMA outperforms in multi-agent benchmarks