PIMAEX: Multi-Agent Exploration through Peer Incentivization

2025-01-02

AI Summary

This work addresses two key challenges in multi-agent reinforcement learning (MARL): insufficient exploration and difficult credit assignment. To this end, we propose PIMAEX, a peer-inspired intrinsic motivation mechanism that jointly models influence and curiosity. Its core innovation lies in the first integration of influence-aware behavioral prediction with curiosity-driven exploration, yielding a scalable, peer-based intrinsic reward function that encourages agents to actively shape peers’ behaviors to discover novel states. Furthermore, we introduce PIMAEX-Communication, a communication-augmented training framework that enhances coordinated exploration under partial observability (e.g., in Consume/Explore environments). Evaluated on tasks featuring deceptive rewards and severe credit assignment difficulties, PIMAEX significantly improves both exploration efficiency and final policy performance, consistently outperforming state-of-the-art baselines across diverse MARL benchmarks.

Technology Category

• Multiagent Systems: Agent Communication
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

While exploration in single-agent reinforcement learning has been studied extensively in recent years, considerably less work has focused on its counterpart in multi-agent reinforcement learning. To address this issue, this work proposes a peer-incentivized reward function inspired by previous research on intrinsic curiosity and influence-based rewards. The extit{PIMAEX} reward, short for Peer-Incentivized Multi-Agent Exploration, aims to improve exploration in the multi-agent setting by encouraging agents to exert influence over each other to increase the likelihood of encountering novel states. We evaluate the extit{PIMAEX} reward in conjunction with extit{PIMAEX-Communication}, a multi-agent training algorithm that employs a communication channel for agents to influence one another. The evaluation is conducted in the extit{Consume/Explore} environment, a partially observable environment with deceptive rewards, specifically designed to challenge the exploration vs. exploitation dilemma and the credit-assignment problem. The results empirically demonstrate that agents using the extit{PIMAEX} reward with extit{PIMAEX-Communication} outperform those that do not.

Problem

• Multi-Robot Learning
• Challenging Environment Exploration
• Credit Assignment

Innovation

• PIMAEX Reward Method
• PIMAEX-Communication Training Method
• Multi-Robot Exploration Efficiency