Zero-Shot Instruction Following in RL via Structured LTL Representations

2025-12-02

arXiv.org

AI Summary

This work addresses the limited zero-shot instruction-following capability of agents in multi-task reinforcement learning when faced with unseen linear temporal logic (LTL) tasks. To overcome this challenge, the authors propose a structured task representation approach that translates LTL specifications into sequences of Boolean formulas constructed via finite automata. A hierarchical neural network is designed to encode both the logical and temporal structure of these formulas, augmented with an attention mechanism to enable reasoning over future subgoals. This representation effectively supports policy conditioning and significantly improves zero-shot generalization and task success rates across multiple complex environments, outperforming existing universal policy methods.

Technology Category

• Multiagent Systems: Multiagent Learning
• Machine Learning: Reinforcement Learning
• Knowledge Representation and Reasoning: Logic Programming

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work

Abstract

We study instruction following in multi-task reinforcement learning, where an agent must zero-shot execute novel tasks not seen during training. In this setting, linear temporal logic (LTL) has recently been adopted as a powerful framework for specifying structured, temporally extended tasks. While existing approaches successfully train generalist policies, they often struggle to effectively capture the rich logical and temporal structure inherent in LTL specifications. In this work, we address these concerns with a novel approach to learn structured task representations that facilitate training and generalisation. Our method conditions the policy on sequences of Boolean formulae constructed from a finite automaton of the task. We propose a hierarchical neural architecture to encode the logical structure of these formulae, and introduce an attention mechanism that enables the policy to reason about future subgoals. Experiments in a variety of complex environments demonstrate the strong generalisation capabilities and superior performance of our approach.

Problem

• zero-shot instruction following
• reinforcement learning
• linear temporal logic
• task generalization
• structured representation

Innovation

• Zero-Shot Instruction Following
• Linear Temporal Logic (LTL)
• Structured Task Representation
• Hierarchical Neural Architecture
• Attention Mechanism