Variational Offline Multi-agent Skill Discovery

2024-05-26

arXiv.org

AI Summary

Extracting emergent subgroup coordination patterns and discovering reusable skills in offline multi-agent settings remains challenging due to the absence of explicit subgroup labels and temporal abstractions. Method: We propose two variational autoencoder frameworks—VO-MASD-3D and VO-MASD-Hier—that jointly model subgroup structure identification and temporal abstraction. They incorporate an unsupervised dynamic grouping mechanism that discovers latent subgroups from agent interactions, and enable end-to-end learning of transferable hierarchical multi-agent skills across offline multi-task domains. Contribution/Results: By integrating graph neural networks, dynamic clustering, and offline reinforcement learning, our approach significantly outperforms existing hierarchical MARL methods on the StarCraft benchmark. It effectively mitigates the sparse and delayed reward problem inherent in offline multi-agent learning and yields skills with zero-shot cross-task transferability.

Technology Category

• Multiagent Systems: Multiagent Learning
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Skills are effective temporal abstractions established for sequential decision making, which enable efficient hierarchical learning for long-horizon tasks and facilitate multi-task learning through their transferability. Despite extensive research, research gaps remain in multi-agent scenarios, particularly for automatically extracting subgroup coordination patterns in a multi-agent task. In this case, we propose two novel auto-encoder schemes: VO-MASD-3D and VO-MASD-Hier, to simultaneously capture subgroup- and temporal-level abstractions and form multi-agent skills, which firstly solves the aforementioned challenge. An essential algorithm component of these schemes is a dynamic grouping function that can automatically detect latent subgroups based on agent interactions in a task. Further, our method can be applied to offline multi-task data, and the discovered subgroup skills can be transferred across relevant tasks without retraining. Empirical evaluations on StarCraft tasks indicate that our approach significantly outperforms existing hierarchical multi-agent reinforcement learning (MARL) methods. Moreover, skills discovered using our method can effectively reduce the learning difficulty in MARL scenarios with delayed and sparse reward signals. The codebase is available at https://github.com/LucasCJYSDL/VOMASD.

Problem

• Automatically extracting subgroup coordination patterns in multi-agent tasks
• Capturing subgroup- and temporal-level abstractions for multi-agent skills
• Transferring discovered subgroup skills across tasks without retraining

Innovation

• Auto-encoder schemes for multi-agent skill discovery
• Dynamic grouping function detects latent subgroups
• Offline multi-task data compatible with skill transfer