State-Centric Decision Process

2026-05-12

AI Summary

Current language environments—such as web browsers and code terminals—lack the explicit state space, observation mappings, verified transitions, and termination criteria required by Markov decision processes. This work proposes a runtime framework enabling agents to actively define task-relevant states through natural language predicates, execute actions to satisfy these predicates, and validate resulting observations, thereby dynamically constructing the core components necessary for structured decision-making. For the first time, this approach allows agents to self-construct state spaces and verified execution trajectories within unstructured linguistic environments, supporting advanced analytical capabilities such as credit assignment, failure localization, progress measurement, and modular replanning. Evaluated across five benchmarks spanning planning, scientific exploration, web-based reasoning, and multi-hop question answering, the method achieves state-of-the-art zero-shot performance, with its advantage markedly increasing as task horizons lengthen.

Abstract

Language environments such as web browsers, code terminals, and interactive simulations emit raw text rather than states, and provide none of the runtime structure that MDP analysis requires. No explicit state space, no observation-to-state mapping, no certified transitions, and no termination criterion. We introduce the State-Centric Decision Process (SDP), a runtime framework that constructs these missing inputs by having the agent build them, predicate by predicate, as it acts. At each step the agent commits to a natural-language predicate describing how the world should look, takes an action to make it true, and checks the observation against it. Predicates that pass become certified states, and the resulting trajectory carries the four objects language environments do not provide, namely a task-induced state space, an observation-to-state mapping, certified transitions, and a termination criterion. We evaluate SDP on five benchmarks spanning planning, scientific exploration, web reasoning, and multi-hop question answering. SDP achieves the best training-free results on all five, with the advantage widening as the horizon grows. The certified trajectories additionally support analyses unavailable to reactive agents, including per-predicate credit assignment, failure localization, partial-progress measurement, and modular operator replacement.

Problem

• language environments
• Markov Decision Process
• state space
• observation-to-state mapping
• termination criterion

Innovation

• State-Centric Decision Process
• language environments
• certified states
• predicate-based reasoning
• MDP construction