Commit to the Bit: Reactive Reinforcement Learning Done Right

2026-05-27

AI Summary

This work addresses the challenge of learning optimal reactive policies in non-Markovian environments or those with limited state features. It proposes Committed Q-learning, a novel variant of Q-learning that operates in finite environments with deterministic observations. The method leverages an action-commitment mechanism—fixing actions under specific state features and resampling only when these features change—combined with state-feature aggregation for policy learning. A key contribution is the introduction of “reconnection robustness,” a condition weaker than the classical $q_\star$-realizability assumption, along with the first formal definition of quasi-Markov environments to enable rigorous theoretical analysis. Under this assumption, the algorithm is proven to converge almost surely to an optimal reactive policy.

Abstract

Reinforcement learning algorithms are commonly analyzed (and designed) under the Markov assumption. This is unrealistic, as most environments encountered in practice are either partially observable, or require function approximation that restricts the agent to access non-Markovian state features. We consider the problem of learning an optimal reactive policy in a finite environment with deterministic observations (or equivalently, hard state aggregation). We introduce a new algorithm, Committed Q-learning, and prove almost-sure convergence to the optimal reactive policy under an intuitive assumption we call rewire-robustness. This assumption is strictly weaker than the $q_\star$-realizability condition used in prior work. Our algorithm is a variant of classical Q-learning in which the behavior policy commits to a single action upon entering a feature, and only resamples actions when the observed feature changes. A crucial part of our analysis is the introduction of quasi-Markov environments.

Problem

• reactive policy
• non-Markovian environment
• hard state aggregation
• deterministic observations
• optimal policy learning

Innovation

• Committed Q-learning
• reactive policy
• rewire-robustness
• quasi-Markov environments
• hard state aggregation