Blockchain-Assisted Demonstration Cloning for Multiagent Deep Reinforcement Learning

2024-03-01

IEEE Internet of Things Journal

AI Summary

To address low sample efficiency, the curse of dimensionality, and insufficient exploration in multi-agent deep reinforcement learning (MDRL), this paper proposes a blockchain-empowered Multi-Expert Demonstration Cloning (MEDC) framework. MEDC innovatively integrates a permissioned blockchain with IPFS to enable trustworthy sharing and end-to-end provenance tracking of expert models; it leverages smart contracts to automatically distribute verified expert policies and augments multi-agent DRL with imitation learning—thereby circumventing malicious model injection risks inherent in federated learning and avoiding local optima induced by reward shaping. Experiments across multiple benchmark tasks demonstrate that MEDC significantly improves convergence speed and robustness, exhibits strong fault tolerance against model failures and adversarial policy injection, and consistently outperforms state-of-the-art federated RL, reward shaping, and imitation-learning-augmented approaches.

Technology Category

• Multiagent Systems: Multiagent Learning
• Machine Learning: Distributed Machine Learning & Federated Learning
• Data Mining & Knowledge Management: Representing, Reasoning, and Using Provenance, Trust

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Security and Privacy: Blockchains and distributed ledgers

Abstract

Multiagent deep reinforcement learning (MDRL) is a promising research area in which agents learn complex behaviors in cooperative or competitive environments. However, MDRL comes with several challenges that hinder its usability, including sample efficiency, curse of dimensionality, and environment exploration. Recent works proposing federated reinforcement learning (FRL) to tackle these issues suffer from problems related to model restrictions and maliciousness. Other proposals using reward shaping (RS) require considerable engineering and could lead to local optima. In this article, we propose a novel Blockchain-assisted multiexpert demonstration cloning (MEDC) framework for MDRL. The proposed method utilizes expert demonstrations in guiding the learning of new MDRL agents, by suggesting exploration actions in the environment. A model sharing framework on Blockchain is designed to allow users to share their trained models, which can be allocated as expert models to requesting users to aid in training MDRL systems. A Consortium Blockchain is adopted to enable traceable and autonomous execution without the need for a single trusted entity. Smart Contracts are designed to manage users and models allocation, which are shared using IPFS. The proposed framework is tested on several applications and is benchmarked against existing methods in FRL, RS, and imitation learning-assisted RL. The results show the outperformance of the proposed framework in terms of learning speed and resiliency to faulty and malicious models.

Problem

• Multi-Agent Deep Reinforcement Learning
• Federated Reinforcement Learning
• Reward Mechanism

Innovation

• Blockchain-Assisted Learning
• Multi-Agent Deep Reinforcement Learning
• Federated Reinforcement Learning Optimization