Expert-Free Online Transfer Learning in Multi-Agent Reinforcement Learning

2025-01-26

AI Summary

Existing multi-agent reinforcement learning (MARL) knowledge transfer approaches rely on expert-designed source policies, offline pretraining, and fixed task boundaries, resulting in poor online adaptability and slow recovery under abrupt task shifts. Method: We propose the first online transfer framework for MARL that requires no expert-provided source policy. It integrates adaptive policy distillation, online importance reweighting, and multi-agent collaborative representation alignment to enable real-time knowledge reuse under dynamic task changes. Results: Evaluated on multiple MARL benchmarks, our method improves convergence speed by 42%, enhances sample efficiency by 3.1×, and reduces policy recovery time after task shifts by 76%. Its core contribution lies in eliminating reliance on pretrained policies and static task assumptions—enabling, for the first time, fully online, expert-free, cross-task continual learning in MARL.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Multiagent Systems: Multiagent Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Reinforcement Learning (RL) enables an intelligent agent to optimise its performance in a task by continuously taking action from an observed state and receiving a feedback from the environment in form of rewards. RL typically uses tables or linear approximators to map state-action tuples that maximises the reward. Combining RL with deep neural networks (DRL) significantly increases its scalability and enables it to address more complex problems than before. However, DRL also inherits downsides from both RL and deep learning. Despite DRL improves generalisation across similar state-action pairs when compared to simpler RL policy representations like tabular methods, it still requires the agent to adequately explore the state-action space. Additionally, deep methods require more training data, with the volume of data escalating with the complexity and size of the neural network. As a result, deep RL requires a long time to collect enough agent-environment samples and to successfully learn the underlying policy. Furthermore, often even a slight alteration to the task invalidates any previous acquired knowledge. To address these shortcomings, Transfer Learning (TL) has been introduced, which enables the use of external knowledge from other tasks or agents to enhance a learning process. The goal of TL is to reduce the learning complexity for an agent dealing with an unfamiliar task by simplifying the exploration process. This is achieved by lowering the amount of new information required by its learning model, resulting in a reduced overall convergence time...

Problem

• Multi-robot Systems
• Reinforcement Learning
• Adaptive Learning

Innovation

• Reinforcement Learning
• Transfer Learning
• Autonomous Learning