Coreset-Based Task Selection for Sample-Efficient Meta-Reinforcement Learning

2025-02-04

AI Summary

Model-agnostic meta-reinforcement learning (MAML-RL) suffers from low sample efficiency due to task redundancy—uniformly utilizing all tasks during meta-training wastes samples on uninformative or highly correlated tasks. Method: We propose a coreset-based task selection method grounded in gradient-space diversity: prior to meta-training, we select a compact, informative subset of tasks exhibiting high inter-task gradient dissimilarity, rather than using the full task distribution uniformly. This is the first application of coreset principles to the task space in meta-RL. Contribution/Results: We theoretically establish that, under general conditions, our method reduces the sample complexity required to achieve an ε-stationary solution to O(1/ε); under a gradient-dominance assumption, this further improves to O(log(1/ε)). By integrating gradient embeddings and weighted coreset construction, we provide convergence analysis within the MAML-LQR framework and empirically validate substantial reductions in sampling requirements and accelerated adaptation across multiple RL benchmarks.

Technology Category

• Machine Learning: Learning with Manifolds
• Multiagent Systems: Multiagent Learning
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

We study task selection to enhance sample efficiency in model-agnostic meta-reinforcement learning (MAML-RL). Traditional meta-RL typically assumes that all available tasks are equally important, which can lead to task redundancy when they share significant similarities. To address this, we propose a coreset-based task selection approach that selects a weighted subset of tasks based on how diverse they are in gradient space, prioritizing the most informative and diverse tasks. Such task selection reduces the number of samples needed to find an $epsilon$-close stationary solution by a factor of O(1/$epsilon$). Consequently, it guarantees a faster adaptation to unseen tasks while focusing training on the most relevant tasks. As a case study, we incorporate task selection to MAML-LQR (Toso et al., 2024b), and prove a sample complexity reduction proportional to O(log(1/$epsilon$)) when the task specific cost also satisfy gradient dominance. Our theoretical guarantees underscore task selection as a key component for scalable and sample-efficient meta-RL. We numerically validate this trend across multiple RL benchmark problems, illustrating the benefits of task selection beyond the LQR baseline.

Problem

• Enhance sample efficiency in meta-RL
• Reduce task redundancy via coreset selection
• Improve adaptation to unseen tasks

Innovation

• Coreset-based task selection
• Diverse tasks in gradient space
• Sample-efficient meta-reinforcement learning