Centralized Reward Agent for Knowledge Sharing and Transfer in Multi-Task Reinforcement Learning

2024-08-20

AI Summary

To address inefficient training and poor generalization caused by sparse rewards in multi-task reinforcement learning, this paper proposes the Centralized Reward Agent (CRA) framework. Methodologically, CRA integrates multi-agent reward shaping, distributed policy training, knowledge distillation, and reward signal propagation—supporting both discrete and continuous action spaces. Its core innovation lies in modeling shaped rewards as transferable knowledge carriers for the first time, enabling cross-task knowledge sharing and transfer via a centralized reward mechanism, thereby facilitating zero-shot and few-shot reward generalization. Evaluated on benchmarks including Meta-World, CRA significantly improves multi-task learning efficiency, generalization performance, and rapid adaptation to unseen tasks. Experimental results validate the effectiveness and robustness of treating rewards as a knowledge representation paradigm.

Technology Category

• Multiagent Systems: Multiagent Learning
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Reward shaping is effective in addressing the sparse-reward challenge in reinforcement learning by providing immediate feedback through auxiliary informative rewards. Based on the reward shaping strategy, we propose a novel multi-task reinforcement learning framework that integrates a centralized reward agent (CRA) and multiple distributed policy agents. The CRA functions as a knowledge pool, which aims to distill knowledge from various tasks and distribute it to individual policy agents to improve learning efficiency. Specifically, the shaped rewards serve as a straightforward metric to encode knowledge. This framework not only enhances knowledge sharing across established tasks but also adapts to new tasks by transferring meaningful reward signals. We validate the proposed method on both discrete and continuous domains, including the representative meta world benchmark, demonstrating its robustness in multi-task sparse-reward settings and its effective transferability to unseen tasks.

Problem

• Addresses sparse-reward challenge in multi-task reinforcement learning
• Proposes centralized reward agent for knowledge sharing across tasks
• Enhances learning efficiency and transferability to unseen tasks

Innovation

• Centralized reward agent for knowledge sharing
• Distilled knowledge via shaped rewards encoding
• Transferable reward signals for new tasks