Semantically Labelled Automata for Multi-Task Reinforcement Learning with LTL Instructions

2026-02-06

AI Summary

This work addresses the challenge of learning a universal policy in multi-task reinforcement learning that generalizes to arbitrary tasks—including previously unseen ones—specified by linear temporal logic (LTL). To this end, the authors propose a novel approach based on semantically annotated automata, which efficiently translates LTL formulas online into structured, semantics-rich state representations and generates highly expressive task embeddings to condition the policy. The method fully supports the expressive power of LTL, enabling semantic enrichment and online construction of automaton states. It achieves state-of-the-art performance across multiple benchmarks and successfully handles complex LTL specifications that are intractable for existing approaches.

Technology Category

• Machine Learning: Reinforcement Learning
• Search and Optimization: Learning to Search
• Planning, Routing, and Scheduling: Learning for Planning and Scheduling

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We study multi-task reinforcement learning (RL), a setting in which an agent learns a single, universal policy capable of generalising to arbitrary, possibly unseen tasks. We consider tasks specified as linear temporal logic (LTL) formulae, which are commonly used in formal methods to specify properties of systems, and have recently been successfully adopted in RL. In this setting, we present a novel task embedding technique leveraging a new generation of semantic LTL-to-automata translations, originally developed for temporal synthesis. The resulting semantically labelled automata contain rich, structured information in each state that allow us to (i) compute the automaton efficiently on-the-fly, (ii) extract expressive task embeddings used to condition the policy, and (iii) naturally support full LTL. Experimental results in a variety of domains demonstrate that our approach achieves state-of-the-art performance and is able to scale to complex specifications where existing methods fail.

Problem

• multi-task reinforcement learning
• linear temporal logic
• task generalization
• universal policy
• LTL specifications

Innovation

• Semantically Labelled Automata
• Multi-Task Reinforcement Learning
• Linear Temporal Logic (LTL)
• Task Embedding
• On-the-Fly Automaton Construction