Select-to-Act: Hierarchical Reinforcement Learning via Adaptive Language Guidance

2026-06-21

AI Summary

Existing instruction-conditioned reinforcement learning methods often overlook the sequential dependencies inherent in natural language instructions, leading to poor sample efficiency. This work proposes HRLLI, a novel framework that introduces a “selection–execution” paradigm through a hierarchical reinforcement learning architecture. HRLLI decomposes language instructions into semantic segments and employs a high-level policy to dynamically select the most relevant segment based on the current state, thereby guiding a low-level policy to execute actions. This adaptive alignment between linguistic guidance and decision-making stages enables more efficient policy learning. Evaluated on the RTFM benchmark, HRLLI significantly outperforms existing approaches, demonstrating that explicitly modeling the dynamic selection of instruction segments is crucial for improving decision efficiency in language-guided reinforcement learning.

Abstract

Reinforcement Learning (RL) has been widely applied to sequential decision-making, yet it often suffers from poor sample efficiency due to costly interactions with the environment. A limited line of recent work has started exploring improving RL efficiency by leveraging external knowledge expressed in natural-language instructions. However, the few existing approaches typically treat the entire instruction as a single conditioning input, failing to account for the stage-dependent nature of language guidance, especially in complex environments. In this paper, we propose \emph{Hierarchical Reinforcement Learning with Language Instructions (HRLLI)}, a hierarchical RL framework that explicitly models natural-language instructions as dynamically selectable semantic guidance during decision-making. HRLLI decomposes instructions into a set of piecewise guidance elements, where each instruction piece may become relevant at different stages of interaction with the environment. A novel hierarchical RL policy structure is then formulated in a \emph{Select-to-Act} paradigm: a high-level semantic policy acts as a guidance selector that selects the most relevant instruction piece to the current state to guide the low-level agent's decision, while a low-level policy executes environment actions conditioned on the selected guidance. The two-level policies are learned simultaneously to maximize augmented expected returns from interactions with the environment. This design enables the agent to adaptively ground language instructions into stage-specific decisions during interaction. Experiments on the instruction-intensive RTFM benchmark show that HRLLI consistently outperforms strong instruction-conditioned RL baselines, demonstrating that explicitly modeling adaptive instruction selection significantly improves the effectiveness of RL.

Problem

• Reinforcement Learning
• Natural Language Instructions
• Hierarchical Reinforcement Learning
• Sample Efficiency
• Sequential Decision-Making

Innovation

• Hierarchical Reinforcement Learning
• Language Guidance
• Select-to-Act
• Instruction Decomposition
• Adaptive Policy