Accelerating Robotic Reinforcement Learning with Agent Guidance

2026-02-12

AI Summary

This work addresses the limited sample efficiency of real-world reinforcement learning and the scalability challenges of existing human-in-the-loop approaches, which are often hampered by operator fatigue and inconsistent guidance. To overcome these limitations, the authors propose the Agent-guided Policy Search (AGPS) framework, which introduces a multimodal agent as an automated supervisor. This agent leverages a semantic world model to provide intrinsic value priors and integrates executable tools, corrective waypoints, and spatial constraints to guide policy search and prune unproductive exploration. Evaluated on precise insertion and deformable object manipulation tasks, AGPS significantly outperforms human-in-the-loop baselines in sample efficiency and achieves scalable, fully autonomous robot reinforcement learning without human intervention.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Multiagent Systems: Adversarial Agents
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Reinforcement Learning (RL) offers a powerful paradigm for autonomous robots to master generalist manipulation skills through trial-and-error. However, its real-world application is stifled by severe sample inefficiency. Recent Human-in-the-Loop (HIL) methods accelerate training by using human corrections, yet this approach faces a scalability barrier. Reliance on human supervisors imposes a 1:1 supervision ratio that limits fleet expansion, suffers from operator fatigue over extended sessions, and introduces high variance due to inconsistent human proficiency. We present Agent-guided Policy Search (AGPS), a framework that automates the training pipeline by replacing human supervisors with a multimodal agent. Our key insight is that the agent can be viewed as a semantic world model, injecting intrinsic value priors to structure physical exploration. By using executable tools, the agent provides precise guidance via corrective waypoints and spatial constraints for exploration pruning. We validate our approach on two tasks, ranging from precision insertion to deformable object manipulation. Results demonstrate that AGPS outperforms HIL methods in sample efficiency. This automates the supervision pipeline, unlocking the path to labor-free and scalable robot learning. Project website: https://agps-rl.github.io/agps.

Problem

• sample inefficiency
• Human-in-the-Loop
• robotic reinforcement learning
• scalability
• supervision bottleneck

Innovation

• Agent-guided Policy Search
• multimodal agent
• semantic world model
• sample efficiency
• robotic reinforcement learning