Zero Shot Coordination for Sparse Reward Tasks with Diverse Reward Shapings

2026-04-27

AI Summary

This work addresses a key limitation in zero-shot coordination (ZSC) within multi-agent reinforcement learning: the common assumption that all agents share an identical reward function, which fails to account for scenarios where partners employ different reward shaping schemes despite pursuing the same objective. To overcome this, the study introduces diverse stochastic reward shaping into the ZSC framework and proposes an ensemble learning approach that trains a universal agent by integrating four distinct reward shaping strategies. Experimental results in the Overcooked environment demonstrate that the proposed method significantly enhances robustness and generalization in zero-shot collaboration, achieving performance gains of 62.2% to 119.2% over baseline ZSC algorithms when cooperating with partners using unseen reward shaping schemes in sparse-reward tasks.

Abstract

Many Multi-Agent Reinforcement Learning (MARL) agents fail to adapt properly to cooperating with agents trained with the same objectives but different seeds, algorithms, or other training differences. This is the problem of Zero-Shot Coordination (ZSC), which focuses on training agents to cooperate well with unknown agents. ZSC has been studied for a variety of tabular cases and simple games such as Hanabi, achieving excellent results. However, existing solutions to ZSC only consider identical rewards for your trained agents and all future partners. This is not realistic for the trained agents, as they do not consider the problem of cooperating with agents that have identical sparse objectives but shape the rewards for those objectives in different manner. To address this issue, we show how to train an ensemble of methods using randomized reward shapings chosen using 4 selection algorithms. Experiments done on the Overcooked environment demonstrate consistent improvements of 62.2%-119.2% in sparse reward over baseline ZSC algorithms when playing with agents that have identical sparse rewards but different reward shapings.

Problem

• Zero-Shot Coordination
• Sparse Reward
• Reward Shaping
• Multi-Agent Reinforcement Learning
• Overcooked

Innovation

• Zero-Shot Coordination
• Reward Shaping
• Sparse Reward
• Multi-Agent Reinforcement Learning
• Ensemble Training