Offline Multi-agent Continual Cooperation via Skill Partition and Reuse

2026-06-24

AI Summary

This work addresses catastrophic forgetting and loss of plasticity in multi-agent offline continuous control, which arise from fixed skill repositories and distributional shifts. To tackle these challenges, the paper proposes COMAD, a novel framework that integrates density estimation–driven identification of skill reusability with a multi-headed policy architecture. COMAD employs an autoencoder to extract coordinated skills from heterogeneous behavioral data and formulates a skill-augmented policy objective that enables dynamic expansion of the skill library. Theoretical analysis provides guarantees for the optimality of skill discovery. Empirical results demonstrate that COMAD significantly outperforms existing methods across multiple multi-agent reinforcement learning benchmarks, effectively mitigating task interference and achieving strong forward and backward transfer capabilities.

Abstract

Extracting skills from multi-agent offline dataset improves learning efficiency via sharing task-invariant coordination skills among tasks. In settings where tasks occur sequentially and the space of skills grows exponentially, existing approaches that rely on heuristically designed and fixed-sized skill libraries struggle to resolve the problem of distributional shift and interference, facing catastrophic forgetting and plasticity loss. To address this problem and endow agents with the ability to continually discover and reuse coordination skills in open-environment, we propose COMAD, a principled framework for Continual Offline Multi-agent Skill Discovery via Skill Partition and Reuse. We first discover skills from mixed multi-agent behavior data with an auto-encoder to transform coordination knowledge into reusable coordination skills. Then we construct a skill-augmented policy learning objective with multi-head architectures, explicitly guiding the advantage function with reusable skills identified via a density-based reusability estimator. Theoretical analysis shows our method approximates the optimum of a continual skill discovery problem. Empirical results across diverse MARL benchmarks show that COMAD continually expands its skill library to mitigate interference, achieving superior forward and backward transfer for task streams compared to multiple baselines.

Problem

• offline multi-agent learning
• continual learning
• skill reuse
• distributional shift
• catastrophic forgetting

Innovation

• Continual Learning
• Offline Multi-agent Reinforcement Learning
• Skill Discovery
• Skill Reuse
• Coordination Skills