Adapting Generalist Robot Policies with Semantic Reinforcement Learning

2026-06-30

AI Summary

This work addresses the challenge of adapting traditional reinforcement learning to out-of-distribution, complex, or long-horizon tasks, where reliance on low-level action spaces hinders effective generalization. To overcome this limitation, the paper introduces Semantic Action Reinforcement Learning (SARL), which leverages learnable language prompts as a semantic action space. SARL enables online optimization of these prompts to modulate pretrained skill compositions within a vision–language–action (VLA) generalist policy. This approach facilitates structured exploration and efficient online adaptation, significantly outperforming existing methods. Empirical results demonstrate that SARL successfully unlocks the capability of general-purpose robotic policies to handle novel tasks in both simulated and real-world environments.

Abstract

Generalist robot policies learn a diverse repertoire of behaviors from large-scale pretraining. In principle, this makes them excellent priors for downstream adaptation via reinforcement learning (RL). In practice, however, standard RL methods leveraging this prior optimize directly over robot actions, requiring the base policy's action distribution to be close to that of a performant policy from the start. This assumption breaks down for complex or long-horizon tasks that fall outside the pretraining distribution. Our key insight is that, for sufficiently expressive generalist policies, language prompts are an effective alternative space for learning to solve such tasks: modulating language inputs elicits skills already within the policy's repertoire, which can be composed to solve tasks beyond its zero-shot capabilities. We propose Semantic Action Reinforcement Learning (SARL), which learns to optimize this prompt space through online interaction, treating the generalist policy as a controllable skill prior. Importantly, leveraging pretrained skills rather than learning new ones from scratch yields structured, semantically meaningful exploration and highly efficient online improvement, and learning to modulate prompts through experience grounds them in induced real-world behaviors for robust task-solving. Across real-world settings and simulated benchmarks, we show SARL unlocks fundamentally new capabilities -- adapting VLA behavior to solve complex, long-horizon tasks -- and significantly outperforms existing approaches for improving robot behavior in deployment.

Problem

• generalist robot policies
• reinforcement learning
• long-horizon tasks
• policy adaptation
• out-of-distribution tasks

Innovation

• Semantic Reinforcement Learning
• Generalist Robot Policies
• Language Prompts
• Skill Composition
• Online Adaptation