StraTA: Incentivizing Agentic Reinforcement Learning with Strategic Trajectory Abstraction

2026-05-07

AI Summary

Existing large language model agents struggle with long-horizon decision-making due to inefficient exploration and poor credit assignment, limiting their ability to optimize long-term task performance. This work proposes the Strategic Trajectory Abstraction (StraTA) framework, which explicitly models trajectory-level policies for the first time. StraTA jointly optimizes policy generation and action execution through a hierarchical GRPO-style rollout mechanism, policy-conditioned action generation, diverse policy sampling, and a critical self-evaluation module. The approach substantially outperforms strong baselines on ALFWorld, WebShop, and SciWorld, achieving success rates of 93.1% and 84.2%, respectively, and attaining a state-of-the-art composite score of 63.5% on SciWorld—surpassing current leading closed-source models.

Abstract

Large language models (LLMs) are increasingly used as interactive agents, but optimizing them for long-horizon decision making remains difficult because current methods are largely purely reactive, which weakens both exploration and credit assignment over extended trajectories. In this work, we present Strategic Trajectory Abstraction (StraTA), a simple framework that introduces an explicit trajectory-level strategy into agentic reinforcement learning (RL). StraTA samples a compact strategy from the initial task state, conditions subsequent actions on that strategy, and trains strategy generation and action execution jointly with a hierarchical GRPO-style rollout design, further enhanced by diverse strategy rollout and critical self-judgment. Experiments on ALFWorld, WebShop, and SciWorld show that StraTA consistently improves both sample efficiency and final performance over strong baselines. StraTA reaches success rates of 93.1% on ALFWorld and 84.2% on WebShop. On SciWorld, StraTA attains a 63.5% overall score, outperforming frontier closed-source models.

Problem

• agentic reinforcement learning
• long-horizon decision making
• trajectory-level strategy
• credit assignment
• exploration

Innovation

• Strategic Trajectory Abstraction
• Agentic Reinforcement Learning
• Hierarchical Policy Learning
• Long-horizon Decision Making
• LLM Agents